MIR-only rlibs

[saethlin]

Status update: The strategy I implemented in this PR is to make builds of non-rlib crates monomorphize everything they need. I was quite thorough about this and had to add a handful of hacks to make it happen.

But I now believe that was a mistake, because of how it breaks proc-macro crates. I think those crates can only work if they are able to cooperate with the compiler correctly to share types across the proc-macro bridge. Currently this is achieved by linking both against the libstd dylib, which contains monomorphizations of the allocator shims and probably a handful of other things. So in order to make this work, I at least need to figure out how to make the compiler expose all the monomorphizations of the standard library, and make the proc-macro crates link against those instead of monomorphizing their own.

---

Originally I was just trying to imitate MIR-only rlibs with -Zcross-crate-inline-threshold=yes, but I think I have enough skill to actually implement the behavior. We shall see.

At time of writing, set RUSTFLAGS=-Zmir-only-libs and if each session that Cargo kicks off only compiles an rlib or an executable, every rlib should contain only MIR and the build should work. Any other configuration probably explodes unceremoniously.

r? @ghost

[saethlin]

@bors try @rust-timer queue

[bors]

⌛ Trying commit d9e0f19 with merge f4568e9...

[bors]

☀️ Try build successful - checks-actions
Build commit: f4568e9 (f4568e9b4c437be978fa03a4b3ebc07f5800c83d)

[rust-timer]

Finished benchmarking commit (f4568e9): comparison URL.

Overall result: ❌✅ regressions and improvements - ACTION NEEDED

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	156.8%	[0.3%, 3121.3%]	166
Regressions ❌ (secondary)	1167.6%	[0.1%, 21506.7%]	116
Improvements ✅ (primary)	-0.8%	[-6.6%, -0.2%]	47
Improvements ✅ (secondary)	-1.8%	[-11.6%, -0.1%]	79
All ❌✅ (primary)	122.0%	[-6.6%, 3121.3%]	213

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	32.5%	[0.4%, 448.7%]	179
Regressions ❌ (secondary)	27.0%	[0.7%, 183.5%]	151
Improvements ✅ (primary)	-2.8%	[-5.1%, -0.5%]	2
Improvements ✅ (secondary)	-8.9%	[-9.2%, -8.6%]	3
All ❌✅ (primary)	32.1%	[-5.1%, 448.7%]	181

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	185.0%	[0.7%, 2991.0%]	151
Regressions ❌ (secondary)	1094.9%	[1.3%, 18459.6%]	114
Improvements ✅ (primary)	-1.6%	[-1.6%, -1.6%]	1
Improvements ✅ (secondary)	-3.5%	[-5.0%, -2.1%]	5
All ❌✅ (primary)	183.8%	[-1.6%, 2991.0%]	152

Binary size

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	73.5%	[0.0%, 423.5%]	134
Regressions ❌ (secondary)	440.4%	[0.1%, 4194.7%]	52
Improvements ✅ (primary)	-18.0%	[-49.7%, -0.8%]	17
Improvements ✅ (secondary)	-32.3%	[-50.4%, -0.4%]	73
All ❌✅ (primary)	63.2%	[-49.7%, 423.5%]	151

Bootstrap: 672.461s -> 1381.575s (105.45%)
Artifact size: 312.46 MiB -> 339.60 MiB (8.69%)

[rustbot]

Failed to set assignee to ghost: invalid assignee

Note: Only org members with at least the repository "read" role, users with write permissions, or people who have commented on the PR may be assigned.

[bors]

☔ The latest upstream changes (presumably #119843) made this pull request unmergeable. Please resolve the merge conflicts.

[oli-obk]

The MCP has been accepted

[saethlin]

👍

This PR is still not really mergeable because it breaks proc macros. I think because I'm monomorphizing the allocator shims wrong.

[bors]

☔ The latest upstream changes (presumably #121644) made this pull request unmergeable. Please resolve the merge conflicts.

[bors]

☔ The latest upstream changes (presumably #122037) made this pull request unmergeable. Please resolve the merge conflicts.

[bors]

☔ The latest upstream changes (presumably #122568) made this pull request unmergeable. Please resolve the merge conflicts.

[rust-log-analyzer]

The job mingw-check-tidy failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

Getting action download info
Download action repository 'msys2/setup-msys2@v2.22.0' (SHA:cc11e9188b693c2b100158c3322424c4cc1dadea)
Download action repository 'actions/checkout@v4' (SHA:b4ffde65f46336ab88eb53be808477a3936bae11)
Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32)
Complete job name: PR - mingw-check-tidy
git config --global core.autocrlf false
shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
---
COPY scripts/sccache.sh /scripts/
RUN sh /scripts/sccache.sh

COPY host-x86_64/mingw-check/reuse-requirements.txt /tmp/
RUN pip3 install --no-deps --no-cache-dir --require-hashes -r /tmp/reuse-requirements.txt \
    && pip3 install virtualenv
COPY host-x86_64/mingw-check/validate-toolstate.sh /scripts/
COPY host-x86_64/mingw-check/validate-error-codes.sh /scripts/

# NOTE: intentionally uses python2 for x.py so we can test it still works.
# NOTE: intentionally uses python2 for x.py so we can test it still works.
# validate-toolstate only runs in our CI, so it's ok for it to only support python3.
ENV SCRIPT TIDY_PRINT_DIFF=1 python2.7 ../x.py test \
           --stage 0 src/tools/tidy tidyselftest --extra-checks=py:lint
# This file is autogenerated by pip-compile with Python 3.10
# by the following command:
#
#    pip-compile --allow-unsafe --generate-hashes reuse-requirements.in
---

#12 [5/8] COPY host-x86_64/mingw-check/reuse-requirements.txt /tmp/
#12 DONE 0.0s

#13 [6/8] RUN pip3 install --no-deps --no-cache-dir --require-hashes -r /tmp/reuse-requirements.txt     && pip3 install virtualenv
#13 0.454   Downloading binaryornot-0.4.4-py2.py3-none-any.whl (9.0 kB)
#13 0.473 Collecting boolean-py==4.0
#13 0.483   Downloading boolean.py-4.0-py3-none-any.whl (25 kB)
#13 0.503 Collecting chardet==5.1.0
---
#13 3.725 Building wheels for collected packages: reuse
#13 3.726   Building wheel for reuse (pyproject.toml): started
#13 4.047   Building wheel for reuse (pyproject.toml): finished with status 'done'
#13 4.048   Created wheel for reuse: filename=reuse-1.1.0-cp310-cp310-manylinux_2_35_x86_64.whl size=181117 sha256=f5f58750481f69515c2c0d1d503daf565e2565c370d07fc6aeb95fe3498b4269
#13 4.048   Stored in directory: /tmp/pip-ephem-wheel-cache-ey3ea7ss/wheels/c2/3c/b9/1120c2ab4bd82694f7e6f0537dc5b9a085c13e2c69a8d0c76d
#13 4.051 Installing collected packages: boolean-py, binaryornot, setuptools, reuse, python-debian, markupsafe, license-expression, jinja2, chardet
#13 4.073   Attempting uninstall: setuptools
#13 4.073     Found existing installation: setuptools 59.6.0
#13 4.074     Not uninstalling setuptools at /usr/lib/python3/dist-packages, outside environment /usr
#13 4.074     Not uninstalling setuptools at /usr/lib/python3/dist-packages, outside environment /usr
#13 4.075     Can't uninstall 'setuptools'. No files were found to uninstall.
#13 4.749 Successfully installed binaryornot-0.4.4 boolean-py-4.0 chardet-5.1.0 jinja2-3.1.2 license-expression-30.0.0 markupsafe-2.1.1 python-debian-0.1.49 reuse-1.1.0 setuptools-66.0.0
#13 4.749 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
#13 5.269 Collecting virtualenv
#13 5.333   Downloading virtualenv-20.25.1-py3-none-any.whl (3.8 MB)
#13 5.564      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.8/3.8 MB 16.5 MB/s eta 0:00:00
#13 5.607 Collecting distlib<1,>=0.3.7
#13 5.617   Downloading distlib-0.3.8-py2.py3-none-any.whl (468 kB)
#13 5.668 Collecting filelock<4,>=3.12.2
#13 5.679   Downloading filelock-3.13.1-py3-none-any.whl (11 kB)
#13 5.679   Downloading filelock-3.13.1-py3-none-any.whl (11 kB)
#13 5.715 Collecting platformdirs<5,>=3.9.1
#13 5.724   Downloading platformdirs-4.2.0-py3-none-any.whl (17 kB)
#13 5.811 Installing collected packages: distlib, platformdirs, filelock, virtualenv
#13 5.972 Successfully installed distlib-0.3.8 filelock-3.13.1 platformdirs-4.2.0 virtualenv-20.25.1
#13 DONE 6.0s

#14 [7/8] COPY host-x86_64/mingw-check/validate-toolstate.sh /scripts/
#14 DONE 0.0s
---
DirectMap4k:      176064 kB
DirectMap2M:     7163904 kB
DirectMap1G:    11534336 kB
##[endgroup]
Executing TIDY_PRINT_DIFF=1 python2.7 ../x.py test            --stage 0 src/tools/tidy tidyselftest --extra-checks=py:lint
+ TIDY_PRINT_DIFF=1 python2.7 ../x.py test --stage 0 src/tools/tidy tidyselftest --extra-checks=py:lint
    Finished `dev` profile [unoptimized] target(s) in 0.03s
##[endgroup]
downloading https://ci-artifacts.rust-lang.org/rustc-builds-alt/a0569fa8f91b5271e92d2f73fd252de7d3d05b9c/rust-dev-nightly-x86_64-unknown-linux-gnu.tar.xz
extracting /checkout/obj/build/cache/llvm-a0569fa8f91b5271e92d2f73fd252de7d3d05b9c-true/rust-dev-nightly-x86_64-unknown-linux-gnu.tar.xz to /checkout/obj/build/x86_64-unknown-linux-gnu/ci-llvm
---
   Compiling tidy v0.1.0 (/checkout/src/tools/tidy)
    Finished `release` profile [optimized] target(s) in 25.72s
##[endgroup]
fmt check
##[error]Diff in /checkout/compiler/rustc_monomorphize/src/collector.rs at line 211:
 use rustc_hir::def::DefKind;
 use rustc_hir::def_id::{DefId, DefIdMap, LocalDefId};
 use rustc_hir::lang_items::LangItem;
+use rustc_middle::middle::codegen_fn_attrs::CodegenFnAttrFlags;
 use rustc_middle::mir::interpret::{AllocId, ErrorHandled, GlobalAlloc, Scalar};
 use rustc_middle::mir::mono::{InstantiationMode, MonoItem};
 use rustc_middle::mir::mono::{InstantiationMode, MonoItem};
##[error]Diff in /checkout/compiler/rustc_monomorphize/src/collector.rs at line 225:
     TypeVisitableExt, VtblEntry,
 use rustc_middle::ty::{GenericArgKind, GenericArgs};
 use rustc_middle::ty::{GenericArgKind, GenericArgs};
-use rustc_middle::{middle::codegen_fn_attrs::CodegenFnAttrFlags};
 use rustc_session::config::CrateType;
 use rustc_session::lint::builtin::LARGE_ASSIGNMENTS;
 use rustc_session::lint::builtin::LARGE_ASSIGNMENTS;
Running `"/checkout/obj/build/x86_64-unknown-linux-gnu/rustfmt/bin/rustfmt" "--config-path" "/checkout" "--edition" "2021" "--unstable-features" "--skip-children" "--check" "/checkout/library/test/src/formatters/terse.rs" "/checkout/library/test/src/formatters/mod.rs" "/checkout/library/test/src/formatters/pretty.rs" "/checkout/library/test/src/formatters/json.rs" "/checkout/library/test/src/test_result.rs" "/checkout/compiler/rustc_pattern_analysis/src/constructor.rs" "/checkout/compiler/rustc_pattern_analysis/src/lints.rs" "/checkout/library/test/src/term/win.rs" "/checkout/compiler/rustc_ast_ir/src/lib.rs" "/checkout/compiler/rustc_ast_ir/src/visit.rs" "/checkout/library/test/src/term/terminfo/searcher/tests.rs" "/checkout/library/test/src/term/terminfo/parm/tests.rs" "/checkout/compiler/rustc_monomorphize/src/partitioning.rs" "/checkout/compiler/rustc_monomorphize/src/polymorphize.rs" "/checkout/compiler/rustc_monomorphize/src/util.rs" "/checkout/compiler/rustc_monomorphize/src/errors.rs" "/checkout/compiler/rustc_monomorphize/src/lib.rs" "/checkout/compiler/rustc_monomorphize/src/collector.rs" "/checkout/library/test/src/term/terminfo/parser/compiled/tests.rs" "/checkout/library/test/src/term/terminfo/parser/compiled.rs" "/checkout/library/test/src/term/terminfo/mod.rs" "/checkout/library/test/src/term/terminfo/searcher.rs" "/checkout/library/test/src/term/terminfo/parm.rs" "/checkout/library/test/src/types.rs" "/checkout/library/test/src/console.rs" "/checkout/library/test/src/bench.rs" "/checkout/library/test/src/stats/tests.rs" "/checkout/library/test/src/lib.rs" "/checkout/library/test/src/term.rs" "/checkout/library/test/src/event.rs" "/checkout/compiler/rustc_driver_impl/src/pretty.rs" "/checkout/compiler/rustc_driver_impl/src/print.rs" "/checkout/compiler/rustc_driver_impl/src/lib.rs" "/checkout/compiler/rustc_driver_impl/src/session_diagnostics.rs" "/checkout/compiler/rustc_driver_impl/src/signal_handler.rs" "/checkout/compiler/rustc_driver_impl/src/args.rs" "/checkout/library/unwind/src/unwinding.rs" "/checkout/library/unwind/src/wasm.rs" "/checkout/library/unwind/src/lib.rs" "/checkout/library/unwind/src/libunwind.rs" "/checkout/compiler/rustc_attr/src/builtin.rs" "/checkout/compiler/rustc_attr/src/lib.rs" "/checkout/compiler/rustc_attr/src/session_diagnostics.rs" "/checkout/library/std/tests/env.rs" "/checkout/library/std/tests/thread.rs" "/checkout/library/std/tests/create_dir_all_bare.rs" "/checkout/library/std/tests/switch-stdout.rs" "/checkout/library/std/tests/run-time-detect.rs" "/checkout/library/std/tests/process_spawning.rs" "/checkout/library/std/tests/common/mod.rs" "/checkout/library/std/build.rs" "/checkout/compiler/rustc_query_system/src/error.rs" "/checkout/library/std/benches/hash/mod.rs" "/checkout/library/std/benches/hash/set_ops.rs" "/checkout/library/std/benches/hash/map.rs" "/checkout/library/std/benches/lib.rs" "/checkout/compiler/rustc_query_system/src/query/mod.rs" "/checkout/compiler/rustc_query_system/src/query/config.rs" "/checkout/compiler/rustc_query_system/src/query/job.rs" "/checkout/compiler/rustc_query_system/src/query/caches.rs" "/checkout/compiler/rustc_query_system/src/query/plumbing.rs" "/checkout/compiler/rustc_query_system/src/lib.rs" "/checkout/compiler/rustc_query_system/src/cache.rs" "/checkout/library/test/src/formatters/junit.rs"` failed.
If you're running `tidy`, try again with `--bless`. Or, if you just want to format code, run `./x.py fmt` instead.
  local time: Thu Mar 21 21:24:00 UTC 2024
  network time: Thu, 21 Mar 2024 21:24:00 GMT
##[error]Process completed with exit code 1.
Post job cleanup.