Add cargo and rust fmt to the toolchain build

[Hecatron]

Hi,
I've been experimenting with my own build of the toolchain for a while now
https://github.com/esp-rs/rust

One of the things I've discovered is a unique bug associated with windows when using a linked / custom toolchain such as esp
Typically some tools such as "cargo make" or "rust-analyser" under visual studio code
tend to run cargo multiple times at the same time (similar to running it multithreaded)

If cargo isn't installed as part of the custom toolchain, then it tends to try and use the cargo native to the host.
But because of the way it does this, it tends to fail under windows when you try to run it more than once at the same time.

• Running multiple cargos simultaneously with linked toolchain under windows rust-lang/rustup#2889

The best workaround I've found is to just install cargo into the custom toolchain

# Fixes VSCode Issues with rust-analyser / cargo-make
x.py build --stage 2 cargo

I also tend to add rustfmt for reformatting code

# Add rustfmt for reformatting code
x.py build --stage 2 rustfmt

Would there be any issues in adding these to the workflows script?
I think it would solve a lot of problems associated with rust-analyser and cargo make which are quite useful.

[Hecatron]

Just to add I've recently been doing a rather big re-write of the github workflows
I've gotten everything split into composite actions to make it easier to handle / read so that the top level workflow isn't massive.
I've also found a workaround for the path issue under windows, and fixed some build related issues as well.
and split the build into multiple jobs to avoid disk space issues.

Currently doing some tweaking for ubuntu then I'll test out macos since it should have the same unix based paths.
It takes around 2.5 hours to do the build so it can take a while to test changes.

[georgik]

rustfmt is being built as part of the toolchain. Let me double check.

[georgik]

@grbd Which toolchain do you use as default?
Former instructions were written in that way that toolchain esp is set as default.
We changed the instructions to leave original default toolchain installed on the system and add option +esp to cargo command when building.
The build command looks like this: cargo +esp build
This command should result in using tools which are not part of custom toolchain directly from default toolchain.

[Hecatron]

I'm using it from a windows platform at the moment.
But it doesn't work if you try and run cargo multiple times at the same time while using a custom toolchain.
(which some dev tools such as rust-analyser, or cargo make do)

I think it's a wider issue with rust / rustup in the way they make cargo visible while using the custom toolchain, in that they try to copy / paste the cargo to a different directory as part of the run which breaks if you're trying to run cargo more than once at the same time since you've got multiple copy / pastes happening at the same time to the same place.
I think it's also only a windows issue.

[georgik]

@grbd I'm running build with esp-rs/rust branch esp-1.57.0.0 - Ubuntu 18 rustc fails on this error:

Copying stage2 rustc from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
thread 'main' panicked at 'fs::read(stamp) failed with No such file or directory (os error 2) ("/home/runner/work/rust-build/rust-build/rust/build/x86_64-unknown-linux-gnu/stage2-rustc/x86_64-unknown-linux-gnu/release/.librustc.stamp")', src/bootstrap/lib.rs:1296:24

https://github.com/esp-rs/rust-build/runs/4394875515?check_suite_focus=true#step:8:7911
Any idea what might be wrong?
The same problem occurs on Windows and macOS.

[Hecatron]

Hmm,
not sure at this point although I think a few things have been updated
The failure seems to be at

Copying stage2 rustc from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)

I think it's searching for the following but can't find it

/home/runner/work/rust-build/rust-build/rust/build/x86_64-unknown-linux-gnu/stage2-rustc/x86_64-unknown-linux-gnu/release/.librustc.stamp

I'll try pulling in the changes to a fork then run it on github workflows

[Hecatron]

I'm running some builds now to see if I can figure out where it is
my suspicion is that it's something ether in the esp rust src itself or something related to the runner

I did spot that you've changed .github/actions/util/checkout-esp-rust/action.yaml
from rust_version to release_version
Originally the way I had it setup was:

• rust_version - the tag / branch checked out from esp-rs/rust such as 1.56.0
• release_version - the name given to the final archive such as 1.56.0.1 (to indicate changes via rust-build)

although perhaps these should be renamed to something more clear

[georgik]

@grbd Thank you very much for testing the build.

Here is explanation of action.yaml change: I've changed one occurrence of rust_version to release_version, because @MabezDev marked new branch as esp-1.57.0.0. In 1.56 it was marked as esp-1.56.0. Having build number at the end matching the release_version is better, because it's easier to recognize changes if multiple build of same Rust toolchain occurs.

Meanwhile I'm building all the versions locally, just to have working reference until we resolve mystery of GitHub runners. Rust builds are pretty complex and we encountered several strange issues like this before.

[Hecatron]

Ok that makes sense.
I'm testing the original build first (just to check github hasn't changed anything)
then I'm going to slowly add the changes in from 1.57 to see where it fails

[georgik]

Hm, the issue seems to be related to .stamp on GitHub Runners.
I can recall one other project which was pushed to repo with timestamp "in-the-future" and GitHub Runner clocks were in different time zone which looks like "past" to cmake, and cmake was stuck in infinite loop of refresh. The solution was to touch all files before running cmake. It's just wild guess. ;-)

[Hecatron]

So I updated my own workflows to be identical to these but using 1.56.0 and that seemed to work fine
then I switched just the version from 1.56.0 to 1.57.0.0 and thats when it failed (on github's native runner)
So the difference I suspect is somewhere within https://github.com/esp-rs/rust

I might try and give this a quick go just to see if it makes any difference
https://github.com/szenius/set-timezone

[georgik]

@MabezDev @jessebraham Any idea which change in Rust 1.57.0 bs 1.56.0 might caused build behavior described above as mentioned by @grbd ?