SSL certificates

[jgm]

Great project!

My cabal.project refers to remote repositories using source-repository-package, but cabal can't fetch them inside this container because the certificates aren't configured.

SSL certificate problem: unable to get local issuer certificate

Is it possible to add certificates to this container so that source-repository-package can be used?

[utdemir]

Hi, thanks!

That is actually a recent addition to this project, have you checked the version v8 (released a week or so ago)? In that one I added the support for SSL certificates, and this test case seemed to work:

ghc-musl/example/cabal.project

Lines 1 to 3 in e923d1e

	source-repository-package
	type: git
	location: git://github.com/utdemir/ghc-musl-test-dep.git

If the v8 also doesn't work, would you be able to provide a repository or a cabal.project snippet which reproduces the issue?

[jgm]

I'm using utdemir/ghc-musl:v8-libgmp-ghc8101 Is that the v8 you mean?

You can reproduce as follows:

git clone -b rc/test-ghc-musl https://github.com/jgm/pandoc tmp-pandoc
cd tmp-pandoc
make debpkg

Results also visible here:
https://github.com/jgm/pandoc/runs/915985898?check_suite_focus=true

Forgive me, I'm a docker ignoramus, so I may be doing things wrong.

[utdemir]

That definitely looks like a bug on this side, and thanks for the test case. I'll take a look at it later today.

[utdemir]

So, I spent some time looking at it. The SSL certificate issue was simple enough, turns out for some reason curl doesn't see the certificates. I'll release a new version with the fix, but before that there is an easy workaround: you can set SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt environment variable which should fix the certificate issue.

However, even after this fix compiling one of the dependencies Diff fails with a linker error:

/nix/store/8fcf4nm3v9jli60hk687z2cgz5dn1gds-binutils-2.31.1/bin/ld: /nix/store/3yxr8wfj 892xs2yxmf4hzbzlrwr5y79h-gcc-9.3.0/lib/gcc/x86_64-unknown-linux-musl/9.3.0/crtbeginT.o: relocation R_X86_64_32 against hidden symbol `__TMC_END__' can not be used when making a shared object
/nix/store/8fcf4nm3v9jli60hk687z2cgz5dn1gds-binutils-2.31.1/bin/ld: /nix/store/3yxr8wfj 892xs2yxmf4hzbzlrwr5y79h-gcc-9.3.0/lib/gcc/x86_64-unknown-linux-musl/9.3.0/crtend.o: relocation R_X86_64_32 against `.ctors' can not be used when making a shared object; recompile with -fPIC
/nix/store/8fcf4nm3v9jli60hk687z2cgz5dn1gds-binutils-2.31.1/bin/ld: final link failed:  nonrepresentable section on output
collect2: error: ld returned 1 exit status `cc' failed in phase `Linker'. (Exit code: 1)

I'll take a look at it later this week. Likely we'll need to pass some flags when building some/all upstream libraries.

Edit: I noticed that inside a fresh, empty container cabal new-install --enable-static --enable-executable-static pandoc finishes successfully, so it seems to me that there is no major issue, we just need to figure out the correct cabal incantation.

[jgm]

Great, thanks for looking into it further.

[utdemir]

I just released v10 which fixes the SSL certificate issue. Now I'm looking at the linker error.

[utdemir]

@jgm I got it compiling! The fix was using cabal's --enable-executable-static flag instead of --ghc-options:

diff --git a/linux/make_artifacts.sh b/linux/make_artifacts.sh
index 9dca400af..e248138ff 100644
--- a/linux/make_artifacts.sh
+++ b/linux/make_artifacts.sh
@@ -16,7 +16,7 @@ ghc --version
 
 cabal v2-update
 cabal v2-clean
-cabal v2-configure --enable-tests -f-export-dynamic -fstatic -fembed_data_files -fbibutils --ghc-options '-optc-Os -optl=-pthread -optl=-static -fPIC -split-sections' . pandoc-citeproc
+cabal v2-configure --enable-tests -f-export-dynamic -fstatic -fembed_data_files -fbibutils --enable-executable-static --ghc-options '-optc-Os -optl=-pthread -split-sections' . pandoc-citeproc
 cabal v2-build . pandoc-citeproc
 cabal v2-test -j1 . pandoc-citeproc
 for f in $(find dist-newstyle -name 'pandoc*' -type f -perm +400); do cp $f /artifacts/; done

Also, a few notes:

• I had to bump the pandoc-citeproc revision since the one tagged in cabal.project did not compile, but you fixed the issue on later revisions.
• The commonmark-hs override on cabal.project is ignored because it has preceding whitespace.
• After the build finishes; make_artifacts.sh script goes on trying to build a deb package, which requires a bunch of tools which doesn't exist inside the container. I'm going to go ahead and add a few which I think would be useful in general case (things like find and xargs), but I am not planning to add things like fakeroot or dpkg-deb to the container. So, my solution would be to get that part of the script to run outside of the container. I hope this works for you.

For reference, here is all the changes I did to get it working: https://gist.github.com/cd6d360059137ed8baa40874f8ca4881

Let me know if there is anything else I can do to help you; I used pandoc countless times and this is the least I can do. Thanks!

[jgm]

That's really great, thanks for tracking this down! I must say, it's very convenient to be able to create the deb in the same container, so without the ability to do that I may end up sticking with my current home-grown system based on alpine, though it's only ghc 8.6.5. I'll have to think about the tradeoffs of moving it outside the container.

[utdemir]

@jgm Fair enough, makes sense. Let me know if you can not find an easier solution, I can put dpkg-deb and fakeroot to the container if there is no better way.

[jgm]

What would be the drawbacks of adding these to the container? (Or adding a way, such as apt or apk, to fetch them?) I notice that not even tar seems to be present. With just a few more tools it would be possible to use this container not just to compile but to package in canonical formats for linux.

[utdemir]

What would be the drawbacks of adding these to the container?

Nothing major, mainly the maintenance cost. For most tools like tar, I have no concerns of adding them. However, more complicated tools sometimes require some work to get it run well inside the container. eg. when I'm try to add the dependencies you need, tar and dpkg-tools worked just fine; however fakeroot did not work straightaway, now I need to figure out what's wrong, add some tests so they won't break again etc.

Or adding a way, such as apt or apk, to fetch them?

This might be a good option to. I could get nix working inside the container, so it should make it possible to install new libraries. Maybe apk would work too.

---

So, in short; there is no major issue other than finding spare time. I guess it would be wisest to spend time on making the container extensible so these things do not end up being a blocker. But, to be honest, this likely won't happen earlier than this weekend, sorry.

[jgm]

That would be great! There's absolutely no urgency on my end, since I already have a working build in a different container.

[utdemir]

@jgm, I released v12 which contains tar, deb and fakeroot. make debpkg now succeeds on the branch you mentioned. However, I noticed that you removed the calls to fakeroot and dpkg, so I haven't tested them. Let me know how it goes.

Regarding to extensibility, apk package manager did not work straightaway inside the container, so it requires some more effort.

[jgm]

That's great news. I'll test right now.

[jgm]

OK, I got a strange error:

dpkg-deb: building package 'pandoc' in '/artifacts/pandoc-2.10.1-1-amd64.deb'.
1990
dpkg-deb (subprocess): compressing tar member: lzma error: Cannot allocate memory
1991
dpkg-deb: error: <compress> from tar -cf subprocess returned error exit status 2
1992
make: *** [debpkg] Error 2

Any ideas?
Here's the full build log, though it's not too informative.
https://github.com/jgm/pandoc/runs/969126050?check_suite_focus=true

[utdemir]

Interesting error. I googled a bit and they tell that higher compression levels might use a lot of memory (if this is indeed caused from a lack of memory). I don't know how much memory GH actions provide, but maybe adjusting these flags might help:

-zcompress-level
           Specify which compression level to use on the compressor backend, when building a package (default is 9 for gzip, 6 for xz).  The accepted values are 0-9
           with: 0 being mapped to compressor none for gzip.  Before dpkg 1.16.2 level 0 was equivalent to compressor none for all compressors.

-Scompress-strategy
           Specify which compression strategy to use on the compressor backend, when building a package (since dpkg 1.16.2). Allowed values are none (since dpkg 1.16.4),
           filtered, huffman, rle and fixed for gzip (since dpkg 1.17.0) and extreme for xz.

-Zcompress-type
           Specify which compression type to use when building a package.  Allowed values are gzip, xz (since dpkg 1.15.6), and none (default is xz).

First, I'd probably try something like -Zgzip -z4.

[jgm]

Strange: compiling pandoc uses a huge amount of memory. It's hard for me to believe that a machine with enough memory to compile pandoc doesn't have enough memory to compress the binary.

But I'll try it with the flags you suggest!

[jgm]

It worked specifying gzip instead of xz (even with level 9 compression).
The resulting deb is quite a bit bigger than before, but I can live with that.
Thanks for all your help with this!

[utdemir]

I'm glad it worked! And I'm sorry it took a bit long, let me know if there's anything else I can do to make things easier.