Switch `run-make` tests from Makefiles to rust

[TimNN]

Rust currently has a suite of run-make tests, which generally test specific rustc invocations or behaviour, or require external tools (eg. grep or nm).

The goal of this issue is to rewrite these run-make tests, which are currently written as Makefiles, in rust to a) get rid of the dependency on external tools such as make and b) make them more accessible to rust contributors by not requiring arcane knowledge of the make tool.

The transition will require at least the following steps:

• Survey the existing run-make tests with regard to what they actually do / which programs they use (and how). I assume the programs will likely fall into one of three categories:
  • Rust tools (rustc, rustdoc)
  • Utilities easily replaceable by rust code, eg. grep
  • Complex utilities, eg. nm, which cannot be easily rewritten in rust
• Design and (partially) implement a support library, which makes the actions identified above easily possible. For example I imagine a test should look something like this:

  extern crate support;
  
  fn main() {
      let s = support::init();
      
      let lib = s.rustc().compile("file1.rs").output_rlib();
      
      assert!(s.nm(lib).filter_lines("some_symbol").count() == 2);
  }

• Add a new mode to compile-test, maybe run-rmake, which runs the new rust-based run-make tests. This will either include compiling the support library, or receiving the support library from a previous build stage.
• Start porting the actual run-make tests, which may involve adding additional functionality to the support library. At that time this issue, or another, will track the state of all the existing tests and include some detailed instruction to allow people to easily contributing by porting one of the existing tests.

---

There are some open questions:

• Should the support library allow execution of arbitrary commands? Limiting commands to only those explicitly added to the support library would mean that there is a single place which lists the external tools we depend on.
• Is the proposed integration with compiletest the correct choice? For example the support library itself may have external dependencies (maybe regex or gcc-rs) which means we should probably use cargo to compile the actual tests. At which point it may be worth considering if compile-test is needed at all or if cargo is enough (Have the support library in src/lib.rs, the new run-make tests in tests/*.rs and auxiliary files in subdirectories of tests/ named after the main test file).

---

If anyone wants to get involved in the process, please leave a comment on this issue or ping me on IRC.

---

Original Issue Description

Based on a short experiment, it looks like rust on msvc only has five build dependencies: Visual Studio, Git, Python, CMake and make, where make is only used for the run-make tests as far as I can tell.

Of those five, the first four are easily installable natively on windows, whereas make wasn't as straight forward to installe when I tried and required msys2 / mingw.

The questions the are:

• Is there any interest in performing such a conversion?
• If so, then which language should those tests be migrated to? Python or Rust seem like the logical choices.
• How should the switch happen? We'll probably want some kind of incremental strategy, since there are quite a few run-make tests.

[petrochenkov]

cc #38565 (comment)

[petrochenkov]

rust on msvc only has five build dependencies: Visual Studio, Git, Python, CMake and make

IIRC, diff was a dependency as well, probably replaceable with git diff

[alexcrichton]

Oh dear I completely forgot about filing an issue to do this!

I would love to have this implementation, I think we should drop make as fast as we can. The test are horribly difficult to write and easy to get wrong whereas I think the equivalent Rust code would be much nicer to read.

I would discourage use of Python for basically the sole reason that "contributors to rust-lang/rust are likely Rust programmers" in the sense that it's far easier for us as a community to maintain Rust code than Python code (empircally this seems true as well).

I agree an incremental strategy is best, and I think that we've got a lot of flexibility in terms of what we move to. In general I think it should look like:

• All tests are basically a main.rs script that's the makefile today.
• Tests can run arbitrary code (this is a crucial part of run-make vs other test suites)
• Tests are given tons of inputs about the compiler and whatnot
• I think that we'll want a support library (e.g. librun_make) or something like that with utility functions (sort of like tools.mk) that all the scripts can link to as well if they'd like to.

Given all that I'd imagine this would be best implemented by extending compiletest slightly to compile a librun_make library and then compile scripts with rustc and then execute them (passing variables as appropriate). That way we could add something like src/test/run-make2 and just slowly start transitioning tests over there.

(note that this'd make a fantastic quest issue to migrate over these tests)

[bstrie]

@alexcrichton Define "quest issue"? Do you mean something like what we did for #35233 ?

[alexcrichton]

Yep! Precisely like that

[TimNN]

I just updated the original issue with a somewhat more detailed plan on going forward, as well as some open questions. Any feedback / thoughts are greatly appreciated.

I will personally start to work on the steps mentioned, although that will have to wait for about two weeks until I finish my bachelor's thesis.

[kennytm]

Some of the run-make tests like crate-data-smoke, codegen-options-parsing and allow-non-lint-warnings-cmdline could be rewritten as ui tests.

Some must still rely on make, mainly dep-info and dep-info-spaces.