Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Debian 2.1 release on Fedora Rawhide #44

Closed
russel opened this issue Jan 16, 2016 · 34 comments
Closed

Debian 2.1 release on Fedora Rawhide #44

russel opened this issue Jan 16, 2016 · 34 comments

Comments

@russel
Copy link
Contributor

russel commented Jan 16, 2016

I picked up the Debian binary, even though I am currently using Fedora Rawhide. I'll reboot to Debian Sid in a minute.

Rawhide has:

/usr/lib64/libbz2.so.1
/usr/lib64/libbz2.so.1.0.6
/usr/lib/libbz2.so.1
/usr/lib/libbz2.so.1.0.6

but not the libbz2.so.1.0 that dstep requires. I hacked a link and used LD_LIBRARY_PATH and I got the help message with an incorrect command line. I then tried a correct command line.

|> LD_LIBRARY_PATH=$HOME/lib.Linux.x86_64 dstep --version /usr/include/libdvbv5/dvb-fe.h
clang version 3.7.0 (tags/RELEASE_370/final)
Target: x86_64-redhat-linux-gnu
Thread model: posix

An unknown error occurred: dstep.core.Exceptions.DStepException@dstep/core/Exceptions.d(13): An unknown error occurred

??:? void dstep.driver.Application.Application.startConversion(immutable(char)[]) [0x589994]
??:? void dstep.driver.Application.Application.run() [0x5892a4]
??:? int dstack.application.Application.Application.releaseStart() [0x59fcf6]
??:? _Dmain [0x58c0b7]
??:? D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ6runAllMFZ9__lambda1MFZv [0x5b9dee]
??:? void rt.dmain2.d_run_main(int, char**, extern (C) int function(char[][])*).tryExec(scope void delegate()) [0x5b9d44]
??:? void rt.dmain2.d_run_main(int, char**, extern (C) int function(char[][])).runAll() [0x5b9daa]
??:? void rt.dmain2.d_run_main(int, char
, extern (C) int function(char[][])
).tryExec(scope void delegate()) [0x5b9d44]
??:? _d_run_main [0x5b9ca1]
??:? main [0x58c151]
??:? __libc_start_main [0x5f0847ff]

dstep.core.Exceptions.DStepException@dstep/core/Exceptions.d(13): An unknown error occurred

??:? void dstep.driver.Application.Application.startConversion(immutable(char)[]) [0x589994]
??:? void dstep.driver.Application.Application.run() [0x5892a4]
??:? int dstack.application.Application.Application.releaseStart() [0x59fcf6]
??:? _Dmain [0x58c0b7]
??:? D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ6runAllMFZ9__lambda1MFZv [0x5b9dee]
??:? void rt.dmain2.d_run_main(int, char**, extern (C) int function(char[][])*).tryExec(scope void delegate()) [0x5b9d44]
??:? void rt.dmain2.d_run_main(int, char**, extern (C) int function(char[][])).runAll() [0x5b9daa]
??:? void rt.dmain2.d_run_main(int, char
, extern (C) int function(char[][])
).tryExec(scope void delegate()) [0x5b9d44]
??:? _d_run_main [0x5b9ca1]
??:? main [0x58c151]
??:? __libc_start_main [0x5f0847ff]

@jacob-carlborg
Copy link
Owner

Hmm, I looks like it doesn't like when both the --version flag and a file is specified.

@jacob-carlborg
Copy link
Owner

I tagged a new version as well, v0.2.2. I think you should be able to try the new version with dub fetch dstep && dub run dstep. This will of course compile DStep.

@russel
Copy link
Contributor Author

russel commented Jan 17, 2016

On Debian Sid, you have to install libclang-3.7-dev and libbz2-dev, yet the build still fails to find -lclang. I am guessing it is not looking in the 3.7 directories, which means the search path is wrong. Debian Sid still defaults to 3.6 and I have not installed the default dev packages, because I want to force 3.7 usage.

|> dub run dstep -- --version
Building package dstep in /home/users/russel/.dub/packages/dstep-0.2.2/
Performing "debug" build using dmd for x86_64.
tango 1.0.3+2.068: target for configuration "static" is up to date.
mambo 0.0.7: target for configuration "library" is up to date.
dstack 0.0.5: target for configuration "library" is up to date.
dstep 0.2.2: building configuration "default"...
.dub/packages/mambo-0.0.7/mambo/util/Traits.d(154,39): Deprecation: typedef is removed
.dub/packages/mambo-0.0.7/mambo/util/Traits.d(182,30): Deprecation: typedef is removed
Linking...
/usr/bin/ld: cannot find -lclang
collect2: error: ld returned 1 exit status
--- errorlevel 1
dmd failed with exit code 1.

@russel
Copy link
Contributor Author

russel commented Jan 17, 2016

On Fedora Rawhide, libclang and libbz2 dependency problems seems solved by having the clang-libs and bzip2-libs packages installed. However, there seems to be a problem in the latest DMD beta release. I am using 2.069 on Sid but 2.070-b1 on Rawhide.

|> dub run dstep -- --version
Building package dstep in /home/russel/.dub/packages/dstep-0.2.2/
Performing "debug" build using dmd for x86_64.
tango 1.0.3+2.068: building configuration "static"...
tango.io.FileScan is deprecated. Please use tango.io.FilePath.toList or tango.io.vfs.* instead.
tango.math.BigInt is deprecated. Please use std.bigint instead.
.dub/packages/tango-1.0.3_2.068/tango/text/xml/SaxParser.d(808,8): Error: class tango.text.xml.SaxParser.SAXException use of object.Throwable.message() is hidden by SAXException; use 'alias message = Throwable.message;' to introduce base class overload set
dmd failed with exit code 1.

@jacob-carlborg
Copy link
Owner

It needs to be able to find libclang. Either it in same directory as DStep or in the standard search paths. I should be the same as any other application. It's quite difficult to take in to considerations all different versions of Linux and their different locations of placing libraries.

@jacob-carlborg
Copy link
Owner

About the beta, it's a known issue [1]. I think it's fixed by this [2].

[1] https://issues.dlang.org/show_bug.cgi?id=15555
[2] dlang/druntime@80e5059

@jacob-carlborg
Copy link
Owner

I could look into linking statically with libclang. I could also set up a docker environment, that would allow me to build for more platforms.

@russel
Copy link
Contributor Author

russel commented Jan 17, 2016

I think the issue is one of allowing the "compiler" to know which CLang to work with. Different Linux distributions have different places to store things.

@jacob-carlborg
Copy link
Owner

Yes, but /usr/lib and /usr/local/lib are standard Posix library search paths.

@russel
Copy link
Contributor Author

russel commented Jan 17, 2016

Indeed, but with Debian they only get populated by the default package if they support multiple versions, as with LLVM and Clang. Whilst Fedora also have a system for distinguishing variants, for LLVM and Clang they only have one. The issue here is to easily enable people to add to the LD_LIBRARY_PATH for the Dub build. From LDC build experience LLVM has a tool for setting all the paths, you just have to point to the version you want.

@jacob-carlborg
Copy link
Owner

The issue here is to easily enable people to add to the LD_LIBRARY_PATH for the Dub build

Wouldn't LD_LIBRARY_PATH=... dub work? Or do you have something else in mind?

From LDC build experience LLVM has a tool for setting all the paths, you just have to point to the version you want

I tried that, didn't work. It didn't read the environment variable for the config binary. But that should hopefully not be needed since in this cases there's only one file that it needs to find, libclang.so.

@russel
Copy link
Contributor Author

russel commented Jan 17, 2016

I'll try the LD_LIBRARY_PATH thing now. Given then context this is probably the easiest way out.

@russel
Copy link
Contributor Author

russel commented Jan 17, 2016

Sadly not :-(

|> find /usr -iname libclang.so
find: ‘/usr/share/doc/google-chrome-stable’: Permission denied
/usr/lib/llvm-3.7/lib/libclang.so
519 anglides:~
|> LD_LIBRARY_PATH=/usr/lib/llvm-3.7/lib dub run dstep -- --version
Building package dstep in /home/users/russel/.dub/packages/dstep-0.2.2/
Performing "debug" build using dmd for x86_64.
tango 1.0.3+2.068: target for configuration "static" is up to date.
mambo 0.0.7: target for configuration "library" is up to date.
dstack 0.0.5: target for configuration "library" is up to date.
dstep 0.2.2: building configuration "default"...
.dub/packages/mambo-0.0.7/mambo/util/Traits.d(154,39): Deprecation: typedef is removed
.dub/packages/mambo-0.0.7/mambo/util/Traits.d(182,30): Deprecation: typedef is removed
Linking...
/usr/bin/ld: cannot find -lclang
collect2: error: ld returned 1 exit status
--- errorlevel 1
dmd failed with exit code 1.

@jacob-carlborg
Copy link
Owner

Do you have a symlink libclang.so pointing to libclang.so.1 or whatever that actual library is called?

@russel
Copy link
Contributor Author

russel commented Jan 17, 2016

Yes. It is created when installing the libclang-.*-dev package:

|> ll /usr/lib/llvm-3.7/lib/libclang.so
lrwxrwxrwx 1 root root 40 Dec 10 14:17 /usr/lib/llvm-3.7/lib/libclang.so -> ../../x86_64-linux-gnu/libclang-3.7.so.1

@jacob-carlborg
Copy link
Owner

Hmm, it looks liked Dub doesn't forward the environment variables. If I invoke DMD directly it works. But it still requires to use LD_LIBRARY_PATH at runtime.

@russel
Copy link
Contributor Author

russel commented Jan 17, 2016

I suppose I could relent and just install the Debian default (which is currently 3.6).

@russel
Copy link
Contributor Author

russel commented Jan 17, 2016

No, that doesn't work. From what I can see Debian install no LLVM related libraries directly in /usr/lib or /lib. LLVM related things are all siloed in /usr/lib/llvm--.

@jacob-carlborg
Copy link
Owner

😞

@russel
Copy link
Contributor Author

russel commented Jan 17, 2016

It appears that Fedora ensure they put an entry for LLVM in a file in ld.so.conf.d so the LLVM libs are in the search path. From what I can tell, Debian do not do this. Fedora allows for multiple LLVMs, so they are siloed, but it is only the most recent and then any past ones applications need. Debian have many, many LLVMs available, but leave it entirely up to clang to sort of dynamic link paths.

@jacob-carlborg
Copy link
Owner

I'm available on IRC, perhaps easier to discuss there.

@russel
Copy link
Contributor Author

russel commented Jan 17, 2016

Until Dub passes LD_LIBRARY_PATH I am not sure there is any solution to building DStep on Debian using Dub. Sad, but…

@jacob-carlborg
Copy link
Owner

I've tried a lots of different combinations of environment variables, but none of them seem to work. The only thing that works is to modify dub.json. Change the -L and -rpath flags in the lflags-posix attribute.

@russel
Copy link
Contributor Author

russel commented Jan 17, 2016

This is seemingly to get worse :-( I have downgraded DMD to 2.069.2 on Fedora. libclang.so is in the path, but libbz2.so is not:

|> ldconfig -v | grep clang
ldconfig: Can't stat /libx32: No such file or directory
ldconfig: Path /usr/lib' given more than once ldconfig: Path/usr/lib64' given more than once
ldconfig: Can't stat /usr/libx32: No such file or directory
libclang.so -> libclang.so

|> ldconfig -v | grep bz2
ldconfig: Can't stat /libx32: No such file or directory
ldconfig: Path /usr/lib' given more than once ldconfig: Path/usr/lib64' given more than once
ldconfig: Can't stat /usr/libx32: No such file or directory
libbz2.so.1 -> libbz2.so.1.0.6
libbz2.so.1 -> libbz2.so.1.0.6

|> dub run dstep -- --version
Building package dstep in /home/russel/.dub/packages/dstep-0.2.2/
Performing "debug" build using dmd for x86_64.
tango 1.0.3+2.068: building configuration "static"...
tango.io.FileScan is deprecated. Please use tango.io.FilePath.toList or tango.io.vfs.* instead.
tango.math.BigInt is deprecated. Please use std.bigint instead.
mambo 0.0.7: building configuration "library"...
.dub/packages/mambo-0.0.7/mambo/serialization/Serializer.d(718,30): Deprecation: typedef is removed
.dub/packages/mambo-0.0.7/mambo/util/Traits.d(154,39): Deprecation: typedef is removed
.dub/packages/mambo-0.0.7/mambo/util/Traits.d(182,30): Deprecation: typedef is removed
dstack 0.0.5: building configuration "library"...
.dub/packages/mambo-0.0.7/mambo/util/Traits.d(154,39): Deprecation: typedef is removed
.dub/packages/mambo-0.0.7/mambo/util/Traits.d(182,30): Deprecation: typedef is removed
dstep 0.2.2: building configuration "default"...
.dub/packages/mambo-0.0.7/mambo/util/Traits.d(154,39): Deprecation: typedef is removed
.dub/packages/mambo-0.0.7/mambo/util/Traits.d(182,30): Deprecation: typedef is removed
Linking...
/usr/bin/ld: cannot find -lclang
/usr/bin/ld: cannot find -lbz2
collect2: error: ld returned 1 exit status

@jacob-carlborg
Copy link
Owner

I don't know what to tell you. libbz2 is a very standard/basic library. If it cannot find that it seems almost broken. Where does Fedora place its libraries?

@russel
Copy link
Contributor Author

russel commented Jan 17, 2016

That was my fault, I forgot to install bzip2-devel. Silly me. Only the -lclang is a real problem, and I do not understand than on Fedora as libclang.so is in the search path.

@jacob-carlborg
Copy link
Owner

Creating a symlink in /usr/lib or /usr/local/lib should work. BTW, is the library the correct architecture?

@russel
Copy link
Contributor Author

russel commented Jan 17, 2016

My current hypothesis is that dmd is driving the linker in such a way as it does not use the ld.so search path.

I will do that just to show it working, but I do not touch /usr/lib or /usr/local/lib manually, that is package territory.

All libraries are x86_64 here.

I manually hacked the final dmd command adding the path and:

dmd -of.dub/packages/dstep-0.2.2/.dub/build/default-debug-linux.posix-x86_64-dmd_2069-FD00298C00987FC76518B1B95E49DF2E/dstep .dub/packages/dstep-0.2.2/.dub/build/default-debug-linux.posix-x86_64-dmd_2069-FD00298C00987FC76518B1B95E49DF2E/dstep.o .dub/packages/dstack-0.0.5/libdstack.a .dub/packages/mambo-0.0.7/libmambo.a .dub/packages/tango-1.0.3_2.068/libtango.a -L--no-as-needed -L-rpath -L. -L-L. -L-L/usr/lib64/llvm -L-lclang -L-lbz2 -g

which works fine. So I think we can conclude it is dmd that is the culprit here.

@jacob-carlborg
Copy link
Owner

You can add -L-L/usr/lib64/llvm to dub.json to avoid manually invoking DMD. You can also append -v to the DMD command to see the linker command it uses.

Is the ld.so search path the same as LD_LIBRARY_PATH?

@russel
Copy link
Contributor Author

russel commented Jan 17, 2016

dstep --help still shows 0.2.1.

LD_LIBRARY_PATH adds to the path that ld.so is the manager of. I now need to work out how dmd can invoke ld without ld.so path being in play. This cannot be.

The next real problem though is how to get commands built with Dub into $PATH without being very, very silly.

@jacob-carlborg
Copy link
Owner

dstep --help still shows 0.2.1.

Looks like I forgot to update that.

LD_LIBRARY_PATH adds to the path that ld.so is the manager of. I now need to work out how dmd can invoke ld without ld.so path being in play. This cannot be.

DMD invokes the C compiler to do the linking. You can see that by adding the -v flag when invoking DMD.

@russel
Copy link
Contributor Author

russel commented Jan 17, 2016

I had a play with this and gcc is the problem, not dmd. Well not per se. It could yet be an option dmd puts out.

I have put out a query with someone who works on GCC on Fedora/RHEL.

@russel
Copy link
Contributor Author

russel commented Jan 17, 2016

I was wrong, ld doesn't use the ldconfig path, only the runtime loader does. This means LLVM shared object location has to be manually specified on both Debian and Fedora. Having to manually edit dstep.json or dstep.sdl in the repository checkout cannot be the solution, just a temporary hack.

So gcc and dmd are working as they should. The problem thus rests with Dub and it needing to provide platform specific settings. Which actually it does, well in the SDL it does, not sure about JSON.

This means it is up to the original DStep source to have all the variants, which means collecting pull requests from volunteers like me. I'll see what I can do tomorrow on this.

@russel
Copy link
Contributor Author

russel commented Jan 18, 2016

Just to note that 2.070-b2 fixes the problem noted earlier regarding 2.070-b1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants