Add GMP Linker Support

[daryakaviani]

I am attempting to use the multi-party-ecdsa library which uses GMP in my Rust lambda function. GMP is a precision arithmetic library which is commonly used in cryptography research, especially via Rust, so its support would be widely useful for the systems, security, and applied cryptography communities.

Running cargo lambda build causes the following error:

error: linking with /Users/daryakaviani/Library/Caches/cargo-zigbuild/0.11.1/zigcc-x86_64-unknown-linux-gnu.sh
...
ld.lld: error: unable to find library -lgmp

This is specific to cargo-lambda's usage of the zigbuild linker because cargo build works successfully when gmp is installed. Also, cargo zigbuild works, so it could be the specific way cargo-lambda handles environment variables in relation to the zigbuild linker. This error is exclusively introduced when I import multi-party-ecdsa in src/main.rs. To confirm that this is a machine-agnostic issue, I tried it on x86-64 Ubuntu, x86-64 Amazon Linux, x86-64 MacOS, and ARM M1 MacOS and received the same results. I attempted a workaround on Linux by disabling the zigbuild linker:

cargo lambda build --disable-zig-linker

cargo lambda deploy \
  --iam-role arn:aws:iam::XXXXXXXXXXXX:role/rust_lambda_test \
  func_name

cargo lambda invoke --remote \
--data-ascii '{"params": "some params"}' \
--output-format json \
func_name

This successfully compiled my artifacts and deployed the function, but invocation caused:

Error: Runtime.ExitError

  × RequestId: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx Error: Runtime exited with error: exit status 1

Is it possible for cargo-zigbuild or cargo-lambda to be updated for GMP support? If there is an existent workaround, please let me know and feel free to move this to Discussions!

@calavera @messense

[calavera]

Also, cargo zigbuild works, so it could be the specific way cargo-lambda handles environment variables in relation to the zigbuild linker

the problem might be that this project is still using an old version of cargo-zigbuild. I took a look at it last weekend, and it would require significant changes because it also requires updating Clap.

[daryakaviani]

@calavera Few questions:

1. Are there plans to update to the latest zigbuild?
2. Meanwhile, what are the other canonical ways to deploy Rust in a lambda cheaply?
3. Would it be possible for me to obtain a sound binary another way and pair that with cargo lambda deploy and cargo lambda invoke?

[calavera]

3. Would it be possible for me to obtain a sound binary another way and pair that with cargo lambda deploy and cargo lambda invoke?

if cargo-zigbuild works for you, you should be able to compile your code with it, then put it in target/lambda/YOUR_PACKAGE_NAME/bootstrap and other cargo lambda subcommands will work.

That won't fix your runtime error which is probably caused because the Lambda environment doesn't have the shared libraries that you need. I cannot help you with that.

[daryakaviani]

other cargo lambda subcommands will work

@calavera This approach did not work for me. Using cargo zigbuild before deploying (even for a simple function that works with cargo lambda build) causes issues. Am I going wrong somewhere?

1. Ran cargo zigbuild --release. This generated an executable with name YOUR_PACKAGE_NAME in the release folder.
2. I copied the executable into the folder target/lambda/YOUR_PACKAGE_NAME and renamed the executable bootstrap.
3. Deployed successfully with cargo deploy.
4. Invokation causes the following error:

Response
{
  "errorType": "Runtime.ExitError",
  "errorMessage": "RequestId: 744d2237-3c02-4240-8405-aef8e90bf9c0 Error: Runtime exited with error: exit status 1"
}

Function Logs
/var/task/bootstrap: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by /var/task/bootstrap)
/var/task/bootstrap: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by /var/task/bootstrap)
/var/task/bootstrap: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by /var/task/bootstrap)
/var/task/bootstrap: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /var/task/bootstrap)
/var/task/bootstrap: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by /var/task/bootstrap)
/var/task/bootstrap: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by /var/task/bootstrap)
/var/task/bootstrap: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by /var/task/bootstrap)
/var/task/bootstrap: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /var/task/bootstrap)

[calavera]

I just released Cargo Lambda 0.13.0 that upgrades cargo-zigbuild to 0.14.1 which should soport GMP.

[daryakaviani]

@calavera Thanks for the update. Unfortunately, this did not seem to fix the problem. cargo zigbuild still works, but after updating to 0.13.0, cargo lambda build still results in:

error: linking with `/home/ubuntu/.cache/cargo-zigbuild/0.14.1/zigcc-x86_64-unknown-linux-gnu.sh` failed: exit status: 1
  |
...
  = note: warning: unsupported linker arg: -znoexecstack
          warning: unsupported linker arg: -zrelro
          warning: unsupported linker arg: -znow
          ld.lld: error: unable to find library -lgmp

Are there linking nuances in relation to the zigbuild?

[calavera]

no difference, it's exactly the same code. Are you sure you're not using docker or some other host to run zigbuild?

          ld.lld: error: unable to find library -lgmp

This error is caused because there is a library missing in your host.

[daryakaviani]

Are you sure you're not using docker or some other host to run zigbuild

@calavera Yep, I am running cargo zigbuild in the same directory as cargo lambda build. To recreate the issue, you can start from your basic lambda example.

In main.rs, import:

use multi_party_ecdsa::protocols::multi_party_ecdsa::gg_2020::state_machine::keygen::Keygen;
use round_based::{StateMachine};

To Cargo.toml, add to dependencies:

round-based = { version = "0.1.4", features = ["dev"] }
multi-party-ecdsa = { version = "0.8.1", git = "https://github.com/ZenGo-X/multi-party-ecdsa.git" }

You should see that in both MacOS and Linux, cargo zigbuild works but cargo lambda build does not work.

[calavera]

I did some investigation and I found why cargo zigbuild doesn't fail, it's basically because it's not really setting Zig as the linker in cases when there is no target provided. So that's the difference, cargo lambda build is setting a target explicitly, and actually using Zig as the linker in all cases.

You can test this by running this command:

cargo zigbuild --target x86_64-unknown-linux-gnu

In theory, you should get the same output than running cargo zigbuild, but it fails like cargo lambda build fails.

I think this might be a problem in Zig itself, it looks very similar to ziglang/zig#8103

@messense do you have any idea how to solve this?

[messense]

Does settings RUSTFLAGS='-L my/lib/directory' env var work?

[messense]

IMO zig might not help with the glibc version issue when linking external dylibs since the GMP shared library might require a higher glibc version.

[calavera]

Adding RUSTFLAGS="-L /usr/lib/x86_64-linux-gnu" seems to do the work, thanks for the hint @messense!

RUSTFLAGS="-L /usr/lib/x86_64-linux-gnu"

I'm closing this issue since it looks like we can compile the code correctly with the library path in the flags.

@daryakaviani as we mentioned earlier, the fact that this code compiles doesn't mean that it will work on Lambda. I don't know if GMP is necessary at runtime, but if it is, you'll have to find a way to bundle it with your binary, or in a layer.