Add missing libblst to GHC options of wrapped-cabal in static mode. #125

KtorZ · 2024-02-07T10:20:24Z

No description provided.

angerman · 2024-02-08T09:40:45Z

@yvan-sraka can you test this? @KtorZ mentioned this did not work 😰

I have this on my list to test tomorrow, but if you can before me that would be great!

yvan-sraka · 2024-02-08T10:02:42Z

I got this issue (is this similar to what you got @KtorZ?):

% nix develop ".#ghc810-static-minimal"
error:
       … while calling the 'derivationStrict' builtin

         at /builtin/derivation.nix:9:12: (source not available)

       … while evaluating derivation 'nix-shell'
         whose name attribute is located at /nix/store/p7iz0r8gs6ppkhj83zjmwyd21k8b7v3y-source/pkgs/stdenv/generic/make-derivation.nix:300:7

       … while evaluating attribute 'nativeBuildInputs' of derivation 'nix-shell'

         at /nix/store/p7iz0r8gs6ppkhj83zjmwyd21k8b7v3y-source/pkgs/stdenv/generic/make-derivation.nix:344:7:

          343|       depsBuildBuild              = lib.elemAt (lib.elemAt dependencies 0) 0;
          344|       nativeBuildInputs           = lib.elemAt (lib.elemAt dependencies 0) 1;
             |       ^
          345|       depsBuildTarget             = lib.elemAt (lib.elemAt dependencies 0) 2;

       (stack trace truncated; use '--show-trace' to show the full trace)

       error: function 'anonymous lambda' called with unexpected argument 'enableShared'

       at /nix/store/fimq5djjixscylfrppd2fh483vz3bgg3-source/overlays/crypto/libblst.nix:1:1:

            1| { stdenv, lib, autoreconfHook, inputs }:
             | ^
            2|

I "fixed" it with this patch:

diff --git a/flake.nix b/flake.nix
index 95a8722..993df4e 100644
--- a/flake.nix
+++ b/flake.nix
@@ -23,7 +23,7 @@
           static-openssl = (final.openssl.override { static = true; });
           static-zlib = final.zlib.override { shared = false; };
           static-pcre = final.pcre.override { shared = false; };
-          static-libblst = (final.libblst.override { enableShared = false; }).overrideDerivation (old: {
+          static-libblst = (final.libblst.override { }).overrideDerivation (old: {
             postFixup = "";
           });
         });

Maybe we should rather update the blst definition in nixpkgs?

KtorZ · 2024-02-08T10:13:43Z

I got this issue (is this similar to what you got @KtorZ?):

% nix develop ".#ghc810-static-minimal"
error:
       … while calling the 'derivationStrict' builtin

         at /builtin/derivation.nix:9:12: (source not available)

       … while evaluating derivation 'nix-shell'
         whose name attribute is located at /nix/store/p7iz0r8gs6ppkhj83zjmwyd21k8b7v3y-source/pkgs/stdenv/generic/make-derivation.nix:300:7

       … while evaluating attribute 'nativeBuildInputs' of derivation 'nix-shell'

         at /nix/store/p7iz0r8gs6ppkhj83zjmwyd21k8b7v3y-source/pkgs/stdenv/generic/make-derivation.nix:344:7:

          343|       depsBuildBuild              = lib.elemAt (lib.elemAt dependencies 0) 0;
          344|       nativeBuildInputs           = lib.elemAt (lib.elemAt dependencies 0) 1;
             |       ^
          345|       depsBuildTarget             = lib.elemAt (lib.elemAt dependencies 0) 2;

       (stack trace truncated; use '--show-trace' to show the full trace)

       error: function 'anonymous lambda' called with unexpected argument 'enableShared'

       at /nix/store/fimq5djjixscylfrppd2fh483vz3bgg3-source/overlays/crypto/libblst.nix:1:1:

            1| { stdenv, lib, autoreconfHook, inputs }:
             | ^
            2|

I "fixed" it with this patch:

diff --git a/flake.nix b/flake.nix
index 95a8722..993df4e 100644
--- a/flake.nix
+++ b/flake.nix
@@ -23,7 +23,7 @@
           static-openssl = (final.openssl.override { static = true; });
           static-zlib = final.zlib.override { shared = false; };
           static-pcre = final.pcre.override { shared = false; };
-          static-libblst = (final.libblst.override { enableShared = false; }).overrideDerivation (old: {
+          static-libblst = (final.libblst.override { }).overrideDerivation (old: {
             postFixup = "";
           });
         });

Maybe we should rather update the blst definition in nixpkgs?

That's exactly what I ran into.

yvan-sraka · 2024-02-08T10:23:20Z

Then, I believe that what you want to do is something more like:

diff --git a/flake.nix b/flake.nix
index 95a8722..438a181 100644
--- a/flake.nix
+++ b/flake.nix
@@ -23,7 +23,8 @@
           static-openssl = (final.openssl.override { static = true; });
           static-zlib = final.zlib.override { shared = false; };
           static-pcre = final.pcre.override { shared = false; };
-          static-libblst = (final.libblst.override { enableShared = false; }).overrideDerivation (old: {
+          static-libblst = final.libblst.overrideDerivation (old: {
+            configureFlags = old.configureFlags ++ [ "--enable-static" "--disable-shared" ];
             postFixup = "";
           });
         });

Do you have a minimal example project, so I can test if it builds as expected?

angerman · 2024-02-08T11:21:27Z

Ahh, so maybe the issue is that we don't pull the blst and friends from IOHK-nix? But rely on nixpkgs upstream? I know that for IOG projects the cryptography folks advised against using nixpkgs releases and default to our vendor release ones.

KtorZ · 2024-02-08T14:04:34Z

Do you have a minimal example project, so I can test if it builds as expected?

Not really minimal but I am experimenting this in kupo. This latest patch seems slightly better but it'll still expect the blst library to be shipped alongside the executable:

otool -L /path/to/kupo:
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.60.1)
        /usr/lib/libz.dylib (compatibility version 1.0.0, current version 1.3.0)
        libblst.dylib (compatibility version 0.0.0, current version 0.0.0)
        /usr/lib/libiconv.dylib (compatibility version 7.0.0, current version 7.0.0)
        /usr/lib/libffi.dylib (compatibility version 10.0.0, current version 10.2.0)

KtorZ · 2024-02-09T16:06:08Z

Fun note 1 (for some definition of fun): if I compile Kupo on my local dev environment, outside of the devx nix-shell, libblst is nowhere to be seen in the linked dependencies. Which is curious (I double-check and I am using the same compiler version on both, and I'd expect the constraints on the cabal.project to yield a same cardano-ledger and plutus versions that either require blst or not).

Fun note 2: when compiling through the devx shell, the linker warns me about "was built for newer macOS version (12.0) than being linked (11.0)"

angerman · 2024-02-10T01:16:01Z

@yvan-sraka can you please try to figure out with @KtorZ why his local dev env supposedly doesn't have/need libblst? That seems odd to me.

plutus-core -> cardano-crypto-class -> libblst

I would expect that dependency to be there?

Side note: Regarding the linker warning, that's not very problematic, it's just that nix passes some deploymen target. I'm fairly sure that warning can be ignored. But would be nice to clean that up as well.

KtorZ · 2024-02-10T11:01:01Z

I would expect that dependency to be there?

I believe it depends on what version of the ledger/plutus gets bundled in. cabal freeze tells me:

             any.plutus-core ==1.15.0.1,
             any.cardano-crypto-class ==2.1.4.0,
             cardano-crypto-class +secp256k1-support,

Which points to: https://github.com/IntersectMBO/cardano-base/blob/a4b01d1b990fab63505f2c20b79ccbbed033f3fc/cardano-crypto-class/cardano-crypto-class.cabal#L115-L125. Could it be that it's an old enough plutus-core package with only foundational work on BLS but unused; thus stripped by GHC somehow? Is the compiler smart-enough to do this kind of thing 😅 ?

angerman · 2024-02-10T13:57:46Z

For dynamic libraries on macOS, we do indeed deadstripping of dynamic libraries.

angerman · 2024-02-10T13:58:48Z

We use the same code paths in nix as well though. Hence I wouldn't expect that to result in any difference.

KtorZ · 2024-02-13T08:23:08Z

I checked and the dependencies selected (cabal freeze) are the same on my local dev env and within the nix shell. The only difference is that I am using:

package cardano-crypto-praos
  flags: -external-libsodium-vrf

on my local setup. Though I don't see why this would impact the need for libblst 🤔. Anyway, I'll be looking into this tomorrow morning with @yvan-sraka

angerman · 2024-02-13T13:40:51Z

Are you certain that flag is only used locally? Or should always be used. Relying on the vrf hack is a bad idea. It should at best be used in development.

The current buildPhase in NixOS/nixpkgs does something like this: ``` ./build.sh ${lib.optionalString stdenv.hostPlatform.isWindows "flavour=mingw64"} ./build.sh -shared ${lib.optionalString stdenv.hostPlatform.isWindows "flavour=mingw64"} ``` So we do build both a dynamic and static version, for convenience, which is then correctly visible in the nix store (both `.a` and `.dylib` are present). BUT, it seems that because of the presence of the dynamic lib, cabal or GHC (?) still chose to bundle the library dynamically despite the static flags. Removing the `.dylib` from the nix store recover the expected behavior (libblst being statically linked). This is odd, but the current fix is to not build the `-shared` version of the library at all when in the static shell. Co-authored-by: yvan-sraka <yvan@sraka.xyz>

yvan-sraka · 2024-02-14T10:14:56Z

flake.nix

          static-zlib = final.zlib.override { shared = false; };
          static-pcre = final.pcre.override { shared = false; };
+          static-libblst = final.libblst.overrideDerivation (old: {
+            configureFlags = old.configureFlags ++ [ "--enable-static" "--disable-shared" ];


@hamishmack IIRC, you encountered a similar issue: even when we explicitly tell Cabal to statically link libblst, it links it dynamically if pkgconfig also declares a dynamic library... So, we override the buildPhase and postFixup to prevent the generation of the .dylib file, and then Cabal behaves as expected... This seems to occur only on macOS? Is there a related Cabal issue already opened?

yvan-sraka · 2024-03-22T07:19:10Z

@KtorZ could you merge CardanoSolutions#1, and then we could fix the tests of this PR :)

Enforce the use of static libraries in `-static` devshell

angerman · 2024-04-04T01:53:15Z

@yvan-sraka I guess this one is good to go? Can you have a last look and then merge it?

KtorZ force-pushed the main branch from df71b3e to b6121b1 Compare February 7, 2024 11:37

yvan-sraka approved these changes Feb 8, 2024

View reviewed changes

yvan-sraka self-assigned this Feb 8, 2024

Add missing libblst to GHC options of wrapped-cabal in static mode.

21af82e

KtorZ force-pushed the main branch from b6121b1 to 21af82e Compare February 8, 2024 13:52

KtorZ added 2 commits February 14, 2024 09:44

comment out post-fixup

2f50808

KtorZ force-pushed the main branch from ea9894f to e693c63 Compare February 14, 2024 09:32

yvan-sraka reviewed Feb 14, 2024

View reviewed changes

Enforce the use of static libraries in -static devshell

8782f65

Merge pull request #1 from yvan-sraka/main

c028f3c

Enforce the use of static libraries in `-static` devshell

yvan-sraka added this pull request to the merge queue Apr 4, 2024

Merged via the queue into input-output-hk:main with commit bb543f0 Apr 4, 2024

Add missing libblst to GHC options of wrapped-cabal in static mode. #125

Add missing libblst to GHC options of wrapped-cabal in static mode. #125

Uh oh!

Conversation

KtorZ commented Feb 7, 2024

Uh oh!

angerman commented Feb 8, 2024

Uh oh!

yvan-sraka commented Feb 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

KtorZ commented Feb 8, 2024

Uh oh!

yvan-sraka commented Feb 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

angerman commented Feb 8, 2024

Uh oh!

KtorZ commented Feb 8, 2024

Uh oh!

KtorZ commented Feb 9, 2024

Uh oh!

angerman commented Feb 10, 2024

Uh oh!

KtorZ commented Feb 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

angerman commented Feb 10, 2024

Uh oh!

angerman commented Feb 10, 2024

Uh oh!

KtorZ commented Feb 13, 2024

Uh oh!

angerman commented Feb 13, 2024

Uh oh!

yvan-sraka Feb 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yvan-sraka commented Mar 22, 2024

Uh oh!

angerman commented Apr 4, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yvan-sraka commented Feb 8, 2024 •

edited

Loading

yvan-sraka commented Feb 8, 2024 •

edited

Loading

KtorZ commented Feb 10, 2024 •

edited

Loading

yvan-sraka Feb 14, 2024 •

edited

Loading