Split release/dist_release makefile rules in two, reorganize and document makefiles

[brson]

What have you changed? (mandatory)

The main thing this patch does is to modify the default cargo release profile to use thinLTO instead of full LTO, and to turn debuginfo off. This reduces full build time by 25%, and partial build time by 30%.

It makes the various release and dist_release rules independent of each other. Now the dist_release rules rely on the scripts/run-cargo.sh script to put a special .cargo/config in place that contains the "real" release profile. This script requires the use of the unstable -Zcargo-profile flag.

One thing to note about this arrangement is that if any of the dist_release commands are interrupted they will leave .cargo/config in place. Subsequent cargo commands print a warning about cargo profiles being unstable, and effectively ignores the config file. I expect this to not be encountered by most contributors as they are not encouraged to run the dist_release targets by hand.

In the process it deletes all of the experimental x-foo makefile rules, leaving only one in place for dist.

As part of this work I restructured and documented the entire makefile. All pre-existing rules remain, they just might be in a different order. The build_release rule, which does the dist release without the extra testing and file moving, is renamed to build_dist_release for consistency, and build_release left in with a note to use build_dist_release.

I have confirmed @BusyJay that even with debuginfo off perf reports symbol names in profiles. Whether this is sufficient for practical profiling I don't know. We'll find out.

Despite weakening the LTO, I expect the runtime performance to still be excellent, and reasonable to use for benchmarking, as long as the comparison is done between two thinLTO builds. That means one should avoid comparing release builds to the actual released binaries; instead always make two release builds to compare, and only as a final sanity check, do dist_release builds.

Locally:

full before
    Finished release [optimized + debuginfo] target(s) in 29m 08s
partial before
    Finished release [optimized + debuginfo] target(s) in 18m 18s
full after
    Finished release [optimized] target(s) in 21m 51s
partial after
    Finished release [optimized] target(s) in 12m 41s

Fixes #4674

What are the type of the changes? (mandatory)

• Improvement (change which is an improvement to an existing feature)

How has this PR been tested? (mandatory)

Scripted timing tests and months of experiments.

Does this PR affect documentation (docs) or release note? (mandatory)

No.

Does this PR affect tidb-ansible update? (mandatory)

No.

Refer to a related PR or issue link (optional)

#4674

Benchmark result if necessary (optional)

See above.

Add a few positive/negative examples (optional)

[siddontang]

Great @brson

Have we already fixed the problem that removing debuginfo from RocksDB?

[brson]

I pushed commits to address feedback @siddontang .

Have we already fixed the problem that removing debuginfo from RocksDB?

No. There are two problems related to debuginfo in RocksDB. They aren't fixed, but they are also not related specifically to the release profile:

• Turn off rocksdb debuginfo in dev mode when tikv debuginfo is off #4711 - In dev mode, RocksDB always has debuginfo on. This is relatively easy to fix by hacking the rocksdb build script, and harder to fix properly in cmake-rs. I intend to fix this soon.

• There isn't a single stable way to enable debuginfo for both rustc and gcc #4894 - There isn't a single way to turn debuginfo back on for both Rust and C/C++. Right now this is addressed by educating developers to use RUSTFLAGS + CXXCFLAGS in the README. I don't know a good technical fix for this.

[siddontang]

@iamxy

please make sure again that now we use dist_release to build TiKV in CI and publish this binary.

[iamxy]

Yep, it has been confirmed.

[siddontang]

I am with this change

PTAL @nrc @breeswish

[Hoverbear]

This looks great, @brson , thanks for organizing this confusing file.

[nrc]

lgtm (using a script to copy the profiles is pretty awful, but I can't think of a better way to do it)