[WIP] --How to handle Windows 32 and 64 bit externals

[Lucarda]

Extracted from @umlaeute @ #38 :

that problem only exists if somebody wants to use both 32bit and 64bit Pds on the same machine.

in theory Pd already has a mechanism to care for this: use a different filename extension for each platform/architecture. e.g. on linux we use .pd_linux instead of the (more standard) .so.

on w32 we could already use .m_i386 instead of the (more standard) .dll. so far nobody has adopted to this. also there currently isn't a .m_amd64 searched for on w64.

so, i think:

• W32 32bit externals should use the .m_i386 extension instead of .dll
• Pd ought to support the .m_amd64 extension for w64
  • for now "W32 64bit" (W64) externals should use .dll as the extension

that's probably a much better scheme than using different paths for different architectures, as it allows a single deken-external to contain multiple architectures.

[katjav]

that's probably a much better scheme than using different paths for different architectures, as it allows a single deken-external to contain multiple architectures.

That could be extended to Linux as well. Also see issue #30. We could set 'alternative extensions' (where available) as the default to facilitate 'fat libraries'. I guess that people will frown upon such a decision though.

[umlaeute]

yes, it should be handled (consistently) on all platforms, at least those that don't natively support fat binaries (everything but OSX).

@katja i'm not sure abut which part of which decision people would frown upon.

[Lucarda]

Although I like the idea of changing file extension I believe we are going to clash with other [external] resources. For example the pthread implementation of Msys2 has the same name for w32 and w64: “libwinpthread-1.dll” and we are going to find same clash with other dependencies.

Eg: Even if we had:

• coll.dll
• coll.m_i386? <-- I beleive in linux is: coll.l_i386
• pthread.dll
• pthread.m_i386

Would be nice to test if a simple rename of a dll extension will effectivly allow this file to be called (windows auto-magic?). Will try to test but I’m expecting negative results.

[umlaeute]

no, you cannot just rename libwinpthread-1.dll to libwinpthread-1.m_i386 (or libwinpthread-1.fudeldudel and expect it to work.

Pd manually does special handling of the extensions, but the windows dependency resolution will not.

[Lucarda]

How horrible will be if Pd adds w64 to the search path.

trying cyclone/coll.dll ...not found
trying cyclone/w64/coll.dll ...succeded

[umlaeute]

very horrible.

[Lucarda]

Obviously
trying cyclone/coll.dll ...not found

is not true as it will be there and it will be found but incompatible. : (

[Lucarda]

Are you sure is "very horrible"?

[umlaeute]

yes: very

[Lucarda]

So I guess users that wants 32 & 64 versions of Pd to coexist should do their appropriate -path -lib thing.

[umlaeute]

well, they could install the 32bit externals into the system-specific path (%CommonProgramFiles%/Pd) and the 64bit externals into the user-specific path (%AppData%\Pd).
this way, they don't have to play with -path or -lib at all.

[Lucarda]

Good solution. Will test asap... : )

[umlaeute]

i'd rather say its a somewhat sane hack (but far from a good solution)

[katjav]

@katja i'm not sure abut which part of which decision people would frown upon.

Quoting @umlaeute as in https://lists.puredata.info/pipermail/pd-dev/2015-02/020082.html:

and the extensions in Pd are already a big mess; any other environment
i know that supports loading of binary plugins, uses the OS's default
extensions: .so, .dylib, .dll; using non-standard extensions is often
very inconvenient...so please do not add another one.

I'm not personally opposed to the 'carnival of extensions' (quoting Miller in the same mailing list thread). It is very useful for projects where you want to supply binaries for 'all' platforms.

[katjav]

• W32 32bit externals should use the .m_i386 extension instead of .dll
• Pd ought to support the .m_amd64 extension for w64
• for now "W32 64bit" (W64) externals should use .dll as the extension

After thinking about this a bit more, the following 'test phase arrangement' came to mind:

• respect .dll as the extension for W32 32bit externals forever
• build new W32 32bit externals with the .m_i386 extension
• build "W32 64bit" (W64) externals with .m_amd64 as the extension
• a .patch file to make Pd W64-ready includes the new extension and PD_LONGINTTYPE thing

This is a relatively safe arrangement with many advantages, most important of which is: no spurious incompatibilities with Pd's and externals already around. I've seen people puzzled and frustrated about Pd-double which was distributed a few years ago with many non-functional externals. This has done more harm than good to Pd's reputation. Especially for the sake of Pd beginners, this history must not be repeated.

The only thing is, such a 'test phase arrangement' runs ahead of Miller's decision about proposed changes.

[umlaeute]

The only thing is, such a 'test phase arrangement' runs ahead of Miller's decision about proposed changes.
which probably makes it futile...

anyhow, the good thing about the 32bit vs 64bit thingee is, that it is simply not possible to load a 32bit dll with a 64bit host - which is somewhat different from the Pd-double experience where externals would load but crash.

also, i hope that the few people will have both 32bit and 64bit installations at the same time, leaving little room for much confusion (though this will most likely not be true in the near future)

[katjav]

which probably makes it futile...

Yes futile as such, but on the other hand consider the following: since Miller doesn't distribute 64 bit binaries for Windows yet, those that will circulate coming months will be 'inofficial' anyhow. To support extension .m_amd64 in an inofficial distribution need not be futile if Miller likes the idea. Even when he does not (yet) accept the pull request, he may want to support this extension in the next release. In other words a 'declaration of intent' would do.

I'm just brainstorming aloud, not saying this would solve all trouble or is worth the effort. Of course anyone can still distribute 64 bit .dll even when .m_amd64 will be supported. And even when all pdlibbuilder users would upgrade to a version using .m_amd64, that leaves the majority of libs untouched as they have other build systems.

[Lucarda]

I'm just brainstorming aloud, not saying this would solve all trouble or is worth the effort.

I had also brainstormed the fact that foo.m_amd64 might need to load pthread.dll or any other compiled lib eg: ogglib.dll, mp3lib.dll or whateverlib.dll. Those dll will clash if they are on the same folder (being impossible to have both.)

I think is better to keep the .dll extension. There are many [externals] that call other dlls and there's no solution for that.

[umlaeute]

small update:

• Pd-0.49 has included the .m_amd64 extension (for Windows; and .l_amd64 on linux,...). so the "test phase arrangment" no longer runs ahead of miller's decisions.

I think is better to keep the .dll extension. There are many [externals] that call other dlls and there's no solution for that.

• i don't think that there are many externals that call others dlls
• dll hell is well known and thoroughly tested

[Lucarda]

i don't think that there are many externals that call others dlls

but there are, so this makes conflicts if 32 and 64 dlls are in the same folder.

I have nothing against .m_amd64, I just tested and it works but it brings no benefits. :(

[Lucarda]

With no benefit I mean we can't ship 32 and 64 externals on the same folder.

we can have:

PDContainer.dll
PDContainer.m_amd64

but here we get stuck as different runtime archs have the same name. :

libgcc_s_seh-1.dll
libstdc++-6.dll
libwinpthread-1.dll

[umlaeute]

after re-thinking this, i think that the only way to resolve multi-arch dependencies in a generic way is to add architecture-specific "dll-paths" to Pd's loading mechanism (with "dll-path" i mean the path searched for dependency dlls of an external; not to be confused with Pd's "search-path")

e.g. when loading an external foo which resides in .../foo/foo.m_l386, Pd could (prior to actually calling LoadLib to load the external) add .../foo/i386/ to the PATH (using SetDllPath).

on Windows, there is nothing that pd-lib-builder can do about it.
instead it would require adaption into Pd (with a longish legacy-support time to not break newer external builds for older Pds).

on linux, we could use -rpath for this (starting right now), and on macOS we could use someting with -install_name (or -R as well).
this could be done in pd-lib-builder (but again: not for Windows).

i'm afraid that by the time an updated Pd that has arch-specific search-paths has come into wide-spread use (it's not going to be Pd-0.49-0; maybe Pd-0.49-1 but more likely Pd-0.50; then add something like 3 years of grace period), the problem will be much less of an issue than it is now.
so any solution might just be over-engineering.

[umlaeute]

With no benefit I mean we can't ship 32 and 64 externals on the same folder.

this is just wrong for >75% of all Windows packages available via deken, as they don't need any external dlls.

of the remaining packages (those with some .dll that are not an external but support libraries), some are just outdated/legacy (oscx, iemxmlrpc) and a number of others require more complicated build-systems than pd-lib-builder can offer (Gem, ofelia, Jamoma, iemmatrix,...).

of the rest, I think about 5 have so far switched to pd-lib-builder (or are likely to do a switch in the near future, like timbreID)) and would actively benefit from a fix for the problem. i doubt that the rest is being held back by concerns about co-installability (more likely, these are simply unmaintained).

so any solution might just be over-engineering.

[Lucarda]

after re-thinking this, i think that the only way to resolve multi-arch dependencies in a generic way is to add architecture-specific "dll-paths" to Pd's loading mechanism (with "dll-path" i mean the path searched for dependency dlls of an external; not to be confused with Pd's "search-path")

e.g. when loading an external foo which resides in .../foo/foo.m_l386, Pd could (prior to actually calling LoadLib to load the external) add .../foo/i386/ to the PATH (using SetDllPath).

on Windows, there is nothing that pd-lib-builder can do about it.
instead it would require adaption into Pd (with a longish legacy-support time to not break newer external builds for older Pds).

This sounds very good for the Windows dll hell.