Fuzzing incremental compilation against crates.io packages

[nikomatsakis]

To really have confidence in incremental compilation, we need to do large-scale testing. One specific way to do this is to fuzz against different versions of crates from crates.io. This corresponds to a very particular, and common, workflow:

• cargo update to get new packages
• cargo build to build with them

The idea here would be to write a tool which iterates over the last N published crates (that is, the last N versions that have been published to crates.io). For each version V that has been published, we would:

• Create a fresh directory
• Load version V-1 (that is, the previous version available on crates.io)
  • if there is no previous version, skip this crate
• Compile version V-1 using -Z incremental=some-temporary-directory
• Load version V in place (overwriting the old files)
• Compile version V using -Z incremental=some-temporary-directory
• Create another directory
• Compile version V from scratch
• Compare the results of the two compilations of version V:
  • they should produce the same errors/warnings
  • if successful, they should produce the same output file
    • or at least it should contain the same symbols etc
    • we'll have to work this out a bit =)

As of this writing, we're not yet at the point where incremental compilation actually works, but we are already tracking dependencies throughout the front-end; just tracking those dependencies sometimes causes ICEs, which means this tool would be of immediate use. Plus, we expect to be tracking dependencies soon.

I imagine that this tool would be written in Rust. Rather than placing it in src/test/tools, it might be better if it lived as its own crate (maybe in rust-lang?), so that it can draw on the full crates.io ecosystem for utility functions and so forth. We would eventually just run it on a regular and continous basis and open issues for any problems that we find.

Since it has relatively few prerequisites, this bug seems like a good "entry vector" for contributing to Rust. I would love to "mentor" someone to work on it.

[mrmiywj]

Hi, Matsakis. It looks fun, I want to work on this. Actually the workflow you described looks clear, I need more details.

1. Maybe just compile one crate with incremental way and common way, then compare the output and code is better? For your provided workflow, it seems that it should compile a whole project with some crates.
2. To compare the outputs, is there any API provided by rustc to generate inner data struct of compile info? If not, maybe comparing the output from stdout and stderr is worth considering.
3. To compare output file, is it possible that the files generated by the two ways is same if compared bit by bit? Sorry that I don't know how rustc incremental compilation works, could you explain it?

[alexcrichton]

FWIW I've got a script which @jonathandturner has also been working with which is essentially "use cargo to iterate over all crates on crates.io and build them", and that may end up being useful here!

[mrmiywj]

@alexcrichton Hi, could you offer the link of the script? It must be helpful!

[alexcrichton]

Certainly! I've pushed it up here -- https://github.com/alexcrichton/cargo-apply

It doesn't have much documentation, and it's kinda a big hack, but the gist is:

• First, modify src/main.rs (near the bottom) to configure how rustc is run over each crate.
• Next cargo run --release
• Wait for output to get populated with a bunch of files
• output/$crate/stdio is the result for each crate.

[nikomatsakis]

Great to see so much interest! I want to point out that there are more opportunities for fuzzing, so whatever we do, it might make sense to try and have some common code with multiple modes. Some other examples I've been thinking about:

• go through git histories and try to compare subsequent commits
• similarly, use git histories to guide random fuzzing; this could be particularly good for detecting that we still report all the errors we should

[cramertj]

It seems as though others have expressed interest already, but I'd enjoy helping out if there's space for one more.

[mrmiywj]

@nikomatsakis
Hi, Niko
These days I'm trying on this. But there are still some issues about this tool I don't understand.

1. Actually what should the tool work on with? Compile a project with Cargo.tom, trying to compile it with dependencies of various version. Or just compile a pkg ?
2. I tried to hack the code provided by @alexcrichton and found that the only difference between result compiled with incremental or not is just a .rlib file.
3. What does load in your workflow mean to? Actually with the tool, I don't need to load any file. Just figure out which crate and version, with ops::compile_pkg will help us compile it.

[mrmiywj]

@nikomatsakis
Hi, I just finish a stereotype of this project here. Could you offer some advices?

1. Does it work correctly? I mean, does it actually produce the incrementally compiled code?
2. What should I do next? How to compare the result of two ways?

Hope for ur advice!

[nikomatsakis]

@mrmiywj

What does load in your workflow mean to? Actually with the tool, I don't need to load any file. Just figure out which crate and version, with ops::compile_pkg will help us compile it.

What I meant by "load" was "download the sources and expand them into a directory". You're right that once we've done that once there isn't really a need to do anything more after that.

Does it work correctly? I mean, does it actually produce the incrementally compiled code?

Hmm. :) Good question! The code looks like it sort of roughly does the right thing, but for incremental mode, it will have to first run the compiler (roughly as you are doing now) with the source from v1 and then run the compiler again with the source from v2 (it seems like you don't currently have the notion of two revisions, just one).

[mrmiywj]

@nikomatsakis
Hi, I updated my repo.
Actually there was something wrong with dup2. Now I think it works. You can read the stdio in incremental and no incremental directory to find that it actually compile incrementally after you run it!

Now the issue is:
How to compile the file and output of the two ways' compilation. Since I read the outputs, I found that they can not be compare word by word. And, how to compare the lib and binary file generated, bit by bit?

[nikomatsakis]

@mrmiywj neat, I will try to take a look tomorrow. I think that the compiler output ought to be comparable word-by-word, perhaps with some filename normalization...is that not the case?

As for compiling the binary generating, that's a trickier problem. I do not believe they will necessarily be binary identical, because there is some freedom permitted to the linker to reorder symbols and the like. We might be able to dump some of the compiler's internal representation (or the LLVM IR) can compare that, though. I'll have to experiment a bit!

[nikomatsakis]

@mrmiywj I am sorry it took me so long to get back to you! I just did a detailed read through your crate and ran some experiments. It does indeed seem quite close.

A few things:

1. The current version seems to simply test that compilation succeeds. That's a good metric, but we could do better -- for example, I think that the output (in terms of errors, warnings etc) ought to be the same when you are building incrementally as when you are building from scratch. But maybe we can add this later. I'd also like to test that the generated binary is "the same" but I'm not sure how to do this in any kind of reliable fashion. (Hmm, maybe we can instrument the compiler to dump the MIR or something internal that we have more control over.)
2. We should (optionally) run the tests too, I think.
3. We probably want either a mode of this tool or some other tool that can automatically generate the list of packages and revisions to test. In other words, I'd like to be able to generate a list of the last 5 versions of each package in crates.io or something and run against all of those, logging any failures. (Perhaps with a blacklist as well.)

[nikomatsakis]

Thinking more on it let's start with the "just check that compilation succeeds" -- we can scour for ICEs and the like. Then we can add on checking the output, running tests, etc. The main thing then is to add some kind of mode so that we can let this thing run over night and come back to a big list of results, I guess.

[nikomatsakis]

(Ideally, it would log its progress or leave notes so that it can be aborted and restart without losing its place. I am imaging just a list of tested items, or perhaps generating a file for each crate and rev with some distinctive name, so you can later check for it to know if the work has already been done.)

[arielb1]

I am doing this as a university project.

[mrmiywj]

@nikomatsakis
Thanks for your reply! I'm sorry that I'll be busy with my finals this week (it's crazy). But I'll continue on this work next week.
Some ideas:

1. I read through the stdout logs, it's comparable. But, line-by-line comparing looks not good. Since the order of compiling packages, the version info are different. Just compare the packages compiled? Or some other ways?
2. Dumping the MIR code is a good idea. But, I do think there maybe some problems. If the ast(or any other internal structure) of MIR objects the two ways are identical, fine, just compare them make check if they are structure identical. If the MIR objects are not identical, which I do think is more likely, the problem looks like, test the two MIR program are identical. It looks like a program verification problem! But, if the create provides test, maybe we could test the two generated binaries? Since I'm not familiar with rustc design, please point out my mistakes.
3. It's not difficult. There is a git repo about crates' information. I can write another tool to complete this.
4. Let's just start with 'check compilation succeeds'! Other features can be added later!

[nikomatsakis]

@arielb1 let's talk, I think we should avoid duplicating effort, but there are lots of kinds of fuzzing to pursue. For example, fuzzing based on git commits or maybe some kind of random changes.

[nikomatsakis]

@mrmiywj

1. Let's just start with 'check compilation succeeds'! Other features can be added later!

Yes.

[nikomatsakis]

So, I started hacking on a related, but different tool:

https://github.com/nikomatsakis/cargo-fozzy/

It basically works now though I plan to make a number of enhancements (you can see the open issues). I'm going to close this issue in favor of enhancing that tool -- if anyone would like to contribute, please check it out!

[nikomatsakis]

Ah, let me leave some notes for posterity. I did some investigation into crates.io metadata but found that it was more difficult than I expected to do the kinds of "incremental update and rebuild/run-tests" than I expected. For example, crates.io will rewrite metadata in the repo which means that, in general, you can't necessarily run tests on packages pulled out from crates.io (at least according to @alexcrichton). In addition, git histories seem to offer a more fine-grained history, so I decided to just tackle that first.

@mrmiywj I wanted to thank you for your energy in producing the initial script :) please check out cargo fozzy!