Running tests with buildRustCrate

[kolloch]

Issue description

buildRustCrate can be used to build binary, library and procmacro crates but I think tests are not yet supported. (I'd be happy if I am wrong!)

I think having separate build tooling for running tests is not optimal:

• I'd like to test the same binaries that I deploy to production.
• Testing with nix would make it potentially easy to provide some native dependencies, like e.g. postgres, and would tie in nicely with providing good nix-shell dependencies.

I'd be interested in the full range of

• "unit tests"
• "doc tests" and
• "integration tests".

It is probably good to separate building tests from running tests.

Pinging some potentially interested users (sorry if you are not interested in this topic): @P-E-Meunier @vitiral @Mic92

I gladly take a full solution but I'd also be interested in pointers in how other languages solve this with nix or and outline on what would be a idiomatic way to achieve this.

[P-E-Meunier]

I'm indeed interested in this topic, but I'm afraid I don't have much time to work on it right now.

If you want to implement it, I'd be happy to review. The way to go is to look at what cargo test does, and try to mimic that. I might be wrong, but I believe it's just as easy as calling rustc just like in the buildPhase, but with option --test.

Then doc tests can be built and run using rustdoc. I would start by editing pkgs/build-support/rust/buildRustCrate/default.nix, and testing it on a tiny example crate (just run carnix generate-nix --src ./. --standalone, and then nix-build Cargo.nix.

I don't think any change needs to be done to Carnix, but don't hesitate to tell me if you need something done in Carnix.

[kolloch]

On thing that is important to me and I would like advice on is having separate derivations for

• A) building the library
• A2) building the binary, depends on A)
• B) building the library with test-config
• C_i=1..n) for each of the different test binaries (depends on B)
• D_i=1..n) running the tests (maybe unnecessary but rerunning the tests should maybe not need a rebuild), depends on C_i

That way, we only do the necessary rustc invocations if something changes during development.

It would still be nice if the derivations could still be created with one function call similarly to buildRustCrate. So the options are:

A) Create a new function that returns a attribute set with a bunch of derivations, keyed by e.g. lib, lib_test, binary.<bin_name>, build_test.<test_name>, run_test.<test_name>, ... This new function could use an extended buildRustCrate under the hood.

B) Make buildRustCrate behave like A with certain parameters (seems not so nice).

C) Make buildRustCrate return a derivation that does all of the above with the right parameters -- having to rebuild, retest everything on change.

I prefer A but I'd like to hear the opinions of more experienced "nixers".

[kolloch]

@andir This is basically done, isn't it?

[andir]

@kolloch yes it kind of is done. buildRustCrate is able to build test tests. I skipped execution of them for now. That can be done with simple pkgs.runCommand invocations as I contributed to crate2nix.

Is there a need to have proper execution in nixpkgs? I think it would be a honorable long-term goal.

[P-E-Meunier]

Oh, I just found out about crate2nix. What was wrong with carnix? How does crate2nix fix it? Are you interested in merging the projects?

[andir]

Oh, I just found out about crate2nix. What was wrong with carnix? How does crate2nix fix it? Are you interested in merging the projects?

While we probably shouldn't have the conversation here but on either of the issue trackers. I'll try to summarize the benefits from my personal perspective (not offensive towards carnix):

• The code generation is a lot saner. It is using an actual templating language to generate the files and not some complex code that generates Nix expressions where you have to take care of manually escaping strings
• The code uses cargo metadata to derive what needs to be done. I do not remember what carnix did but I can now just use cargo metadata | jq . and look at the metadata that is used to build the Nix file.
• It had tests that I could easily extend whenever I added something
• @kolloch is super responsive to feedback on the issue tracker and generally very nice to work with <3
• lastly it was much more approachable to contributions and it had a some documentation on the README.md.

[kolloch]

@P-E-Meunier : I'd love to chat but I have trouble contacting you. I send you a message on lobste.rs and tried to do the same on IRC. Email would be terribly convenient for me.