Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid declaring a fake dependency edge #68298

Merged
merged 2 commits into from Jan 23, 2020

Conversation

@Mark-Simulacrum
Copy link
Member

Mark-Simulacrum commented Jan 17, 2020

When we're producing an rlib, we do not need anything more than an rmeta file
for each of our dependencies (this is indeed utilized by Cargo for pipelining).
Previously, we were still storing the paths of possible rlib/dylib crates, which
meant that they could still plausibly be accessed. With -Zbinary-dep-depinfo,
that meant that Cargo thought that rustc was using both the rlib and an (earlier
emitted) rmeta, and so needed a recompile, as the rlib may have finished writing
after compilation started (for more detail, see issue 68149).

This commit changes metadata loading to not store the filepaths of dylib/rlib if
we're going to end up creating an rlib only.

Fixes #68149.

@Mark-Simulacrum

This comment has been minimized.

Copy link
Member Author

Mark-Simulacrum commented Jan 17, 2020

To be honest I'm not super happy with this change - feels a bit ad-hoc and I worry it could change under us and we wouldn't notice. Not sure we can do better though.

cc @petrochenkov as well

@petrochenkov petrochenkov self-assigned this Jan 17, 2020
@ehuss

This comment has been minimized.

Copy link
Contributor

ehuss commented Jan 17, 2020

Can you explain more how changing the order the files are printed should fix the issue?

@Mark-Simulacrum

This comment has been minimized.

Copy link
Member Author

Mark-Simulacrum commented Jan 17, 2020

Oh, I was over eager in a refactor, the code should call next() on the iterator instead of pushing the whole iterator on.

@Mark-Simulacrum Mark-Simulacrum force-pushed the Mark-Simulacrum:binary-depdep-fix branch from 1b047ff to 9457681 Jan 17, 2020
@Mark-Simulacrum

This comment has been minimized.

Copy link
Member Author

Mark-Simulacrum commented Jan 17, 2020

Fixed.

@alexcrichton

This comment has been minimized.

Copy link
Member

alexcrichton commented Jan 17, 2020

I think an alternate solution, which may be more of an "at the source" fix, might be to avoid reading these files entirely? For example if we find *.rmeta and *.rlib for a dependency, and we're producing an rlib, then in theory rustc shouldn't even touch the *.rlib. In rustc's own internal data structures we could avoid even listing the path to the *.rlib.

I feel like this sort of route would protect against potential bugs where we don't say we depend on the rlib but rustc still has access to it and could accidentally use it. I'd prefer if we simply don't give rustc access to the rlib internally by not listing the path for the rlib if we can.

@Mark-Simulacrum

This comment has been minimized.

Copy link
Member Author

Mark-Simulacrum commented Jan 17, 2020

I can try to experiment with that path -- I think @petrochenkov planned on trying to do so based on comments on the issue. I do think it's better :)

@Mark-Simulacrum Mark-Simulacrum force-pushed the Mark-Simulacrum:binary-depdep-fix branch 2 times, most recently from 12bdd0e to 90ba750 Jan 18, 2020
@Mark-Simulacrum

This comment has been minimized.

Copy link
Member Author

Mark-Simulacrum commented Jan 18, 2020

Initial implementation for the proposed "at the source" change is up, though I've not run tests locally.

@rust-highfive

This comment has been minimized.

Copy link
Collaborator

rust-highfive commented Jan 18, 2020

The job x86_64-gnu-llvm-7 of your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
2020-01-18T19:29:56.4643708Z ========================== Starting Command Output ===========================
2020-01-18T19:29:56.4662790Z [command]/bin/bash --noprofile --norc /home/vsts/work/_temp/0b517f8c-e5c2-4c3b-a661-43bfeaa88f9e.sh
2020-01-18T19:29:56.6331257Z 
2020-01-18T19:29:56.6377464Z ##[section]Finishing: Disable git automatic line ending conversion
2020-01-18T19:29:56.6383856Z ##[section]Starting: Checkout rust-lang/rust@refs/pull/68298/merge to s
2020-01-18T19:29:56.6385444Z Task         : Get sources
2020-01-18T19:29:56.6385883Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
2020-01-18T19:29:56.6385923Z Version      : 1.0.0
2020-01-18T19:29:56.6385956Z Author       : Microsoft
---
2020-01-18T19:29:57.6006464Z ##[command]git remote add origin https://github.com/rust-lang/rust
2020-01-18T19:29:57.6019733Z ##[command]git config gc.auto 0
2020-01-18T19:29:57.6021811Z ##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
2020-01-18T19:29:57.6024116Z ##[command]git config --get-all http.proxy
2020-01-18T19:29:57.6030690Z ##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/68298/merge:refs/remotes/pull/68298/merge
---
2020-01-18T20:27:38.0431546Z .................................................................................................... 1700/9539
2020-01-18T20:27:45.7619423Z .................................................................................................... 1800/9539
2020-01-18T20:27:56.9751472Z .................i.................................................................................. 1900/9539
2020-01-18T20:28:04.5207034Z .................................................................................................... 2000/9539
2020-01-18T20:28:20.6595416Z .......iiiii........................................................................................ 2100/9539
2020-01-18T20:28:30.3783369Z .................................................................................................... 2300/9539
2020-01-18T20:28:32.9491442Z .................................................................................................... 2400/9539
2020-01-18T20:28:38.6450807Z .................................................................................................... 2500/9539
2020-01-18T20:28:59.9587108Z .................................................................................................... 2600/9539
---
2020-01-18T20:31:49.0425586Z ....................................................i...............i............................... 4900/9539
2020-01-18T20:31:57.5893591Z .................................................................................................... 5000/9539
2020-01-18T20:32:05.6649943Z ...............................................................................................i.... 5100/9539
2020-01-18T20:32:11.1645122Z .................................................................................................... 5200/9539
2020-01-18T20:32:22.5807141Z ...................................................................ii.ii...........i................ 5300/9539
2020-01-18T20:32:32.4576552Z ....i............................................................................................... 5500/9539
2020-01-18T20:32:43.0757976Z .................................................................................................... 5600/9539
2020-01-18T20:32:49.9716357Z .....................................................i.............................................. 5700/9539
2020-01-18T20:32:57.3103487Z .................................................................................................... 5800/9539
2020-01-18T20:32:57.3103487Z .................................................................................................... 5800/9539
2020-01-18T20:33:08.0393809Z .................................................................................................... 5900/9539
2020-01-18T20:33:15.3735031Z ............................................ii...i..ii...........i.................................. 6000/9539
2020-01-18T20:33:39.1082060Z .................................................................................................... 6200/9539
2020-01-18T20:33:47.8643389Z .................................................................................................... 6300/9539
2020-01-18T20:33:47.8643389Z .................................................................................................... 6300/9539
2020-01-18T20:33:54.6288374Z ........................................................................i..ii....................... 6400/9539
2020-01-18T20:34:24.7078710Z .................................................................................................... 6600/9539
2020-01-18T20:34:27.9964411Z ................................................i................................................... 6700/9539
2020-01-18T20:34:30.3628112Z .................................................................................................... 6800/9539
2020-01-18T20:34:32.8900271Z ................................................i................................................... 6900/9539
---
2020-01-18T20:36:16.1520084Z .................................................................................................... 7500/9539
2020-01-18T20:36:21.2220886Z .................................................................................................... 7600/9539
2020-01-18T20:36:27.5025521Z .................................................................................................... 7700/9539
2020-01-18T20:36:34.7683426Z .................................................................................................... 7800/9539
2020-01-18T20:36:46.5345454Z .....................................F.............................................................i 7900/9539
2020-01-18T20:36:53.1479079Z iiiiii.............................................................................................. 8000/9539
2020-01-18T20:37:09.5199863Z .................................................................................................... 8200/9539
2020-01-18T20:37:21.7809800Z .................................................................................................... 8300/9539
2020-01-18T20:37:35.1463887Z .................................................................................................... 8400/9539
2020-01-18T20:37:41.3164272Z .................................................................................................... 8500/9539
---
2020-01-18T20:39:43.6799428Z ---- [ui] ui/rmeta-rpass.rs stdout ----
2020-01-18T20:39:43.6800351Z 
2020-01-18T20:39:43.6801117Z error: test compilation failed although it shouldn't!
2020-01-18T20:39:43.6801415Z status: exit code: 1
2020-01-18T20:39:43.6802408Z command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/rmeta-rpass.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-Zdeduplicate-diagnostics=no" "-C" "prefer-dynamic" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/rmeta-rpass/a" "-Crpath" "-O" "-Cdebuginfo=0" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/rmeta-rpass/auxiliary"
2020-01-18T20:39:43.6803465Z ------------------------------------------
2020-01-18T20:39:43.6803746Z 
2020-01-18T20:39:43.6804219Z ------------------------------------------
2020-01-18T20:39:43.6804497Z stderr:
2020-01-18T20:39:43.6804497Z stderr:
2020-01-18T20:39:43.6804994Z ------------------------------------------
2020-01-18T20:39:43.6805302Z error[E0560]: struct `rmeta_aux::Foo` has no field named `field`
2020-01-18T20:39:43.6806110Z    |
2020-01-18T20:39:43.6806340Z LL |     let _ = Foo { field: 42 };
2020-01-18T20:39:43.6806602Z    |                   ^^^^^ help: a field with a similar name exists: `field2`
2020-01-18T20:39:43.6806817Z 
---
2020-01-18T20:39:43.6842226Z thread 'main' panicked at 'Some tests failed', src/tools/compiletest/src/main.rs:387:22
2020-01-18T20:39:43.6843598Z note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
2020-01-18T20:39:43.6849528Z 
2020-01-18T20:39:43.6849607Z 
2020-01-18T20:39:43.6855511Z command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib" "--rustc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "--src-base" "/checkout/src/test/ui" "--build-base" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui" "--stage-id" "stage2-x86_64-unknown-linux-gnu" "--mode" "ui" "--target" "x86_64-unknown-linux-gnu" "--host" "x86_64-unknown-linux-gnu" "--llvm-filecheck" "/usr/lib/llvm-7/bin/FileCheck" "--host-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--target-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--docck-python" "/usr/bin/python2.7" "--lldb-python" "/usr/bin/python2.7" "--gdb" "/usr/bin/gdb" "--quiet" "--llvm-version" "7.0.0\n" "--system-llvm" "--cc" "" "--cxx" "" "--cflags" "" "--llvm-components" "" "--llvm-cxxflags" "" "--adb-path" "adb" "--adb-test-dir" "/data/tmp/work" "--android-cross-path" "" "--color" "always"
2020-01-18T20:39:43.6864454Z 
2020-01-18T20:39:43.6864514Z 
2020-01-18T20:39:43.6870477Z failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
2020-01-18T20:39:43.6870551Z Build completed unsuccessfully in 1:03:33
2020-01-18T20:39:43.6870551Z Build completed unsuccessfully in 1:03:33
2020-01-18T20:39:43.6934632Z == clock drift check ==
2020-01-18T20:39:43.6963926Z   local time: Sat Jan 18 20:39:43 UTC 2020
2020-01-18T20:39:43.8552078Z   network time: Sat, 18 Jan 2020 20:39:43 GMT
2020-01-18T20:39:43.8558160Z == end clock drift check ==
2020-01-18T20:39:44.4218625Z 
2020-01-18T20:39:44.4316704Z ##[error]Bash exited with code '1'.
2020-01-18T20:39:44.4332282Z ##[section]Finishing: Run build
2020-01-18T20:39:44.4353605Z ##[section]Starting: Checkout rust-lang/rust@refs/pull/68298/merge to s
2020-01-18T20:39:44.4355441Z Task         : Get sources
2020-01-18T20:39:44.4355491Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
2020-01-18T20:39:44.4355557Z Version      : 1.0.0
2020-01-18T20:39:44.4355599Z Author       : Microsoft
2020-01-18T20:39:44.4355599Z Author       : Microsoft
2020-01-18T20:39:44.4355647Z Help         : [More Information](https://go.microsoft.com/fwlink/?LinkId=798199)
2020-01-18T20:39:44.4355718Z ==============================================================================
2020-01-18T20:39:44.8832126Z Cleaning any cached credential from repository: rust-lang/rust (GitHub)
2020-01-18T20:39:44.8873306Z ##[section]Finishing: Checkout rust-lang/rust@refs/pull/68298/merge to s
2020-01-18T20:39:44.8994819Z Cleaning up task key
2020-01-18T20:39:44.8995616Z Start cleaning up orphan processes.
2020-01-18T20:39:44.9116412Z Terminate orphan process: pid (3497) (python)
2020-01-18T20:39:44.9392970Z ##[section]Finishing: Finalize Job

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@petrochenkov

This comment has been minimized.

Copy link
Contributor

petrochenkov commented Jan 18, 2020

This is what I wanted to do, basically.
Except I wanted to write something like self.sess.crate_types.borrow().all(|crate_type| crate_type == CrateType::Rlib || crate_type == CrateType::Rmeta), but it turns out there's no CrateType::Meta.

@petrochenkov

This comment has been minimized.

Copy link
Contributor

petrochenkov commented Jan 18, 2020

I've just realized that if the dependency is a proc macro, then in addition to its rmeta we also need its dylib early, for dlopening. That makes it a dependency, in theory.

(Same applies to plugin dependencies, but -Zbinary-dep-depinfo doesn't produce depinfo for them at all (which is incorrect, but nobody cares).)

@Mark-Simulacrum Mark-Simulacrum mentioned this pull request Jan 19, 2020
0 of 1 task complete
@Mark-Simulacrum

This comment has been minimized.

Copy link
Member Author

Mark-Simulacrum commented Jan 19, 2020

I will refactor/correct the code as you've suggested.

I believe it is indeed the case that we don't care about plugin dependencies (yet, at least). I will not try to solve that problem in this PR.

@Mark-Simulacrum Mark-Simulacrum force-pushed the Mark-Simulacrum:binary-depdep-fix branch from 90ba750 to 0f9f382 Jan 20, 2020
@Mark-Simulacrum

This comment has been minimized.

Copy link
Member Author

Mark-Simulacrum commented Jan 20, 2020

The failing test case (https://github.com/rust-lang/rust/blob/master/src/test/ui/rmeta-rpass.rs) appears to assert that when we find an rmeta and rlib in the linker path, we should prefer the rlib. I'm not sure why this is true.

Per #10786, we would generally want this to be an error (as the metadata in the two files is incompatible), but I don't think we should load both -- I consider it in an unstable implementation detail which file is ultimately used if you have incompatible files.

@rust-highfive

This comment has been minimized.

Copy link
Collaborator

rust-highfive commented Jan 20, 2020

The job x86_64-gnu-llvm-7 of your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
2020-01-20T16:49:30.1992150Z ========================== Starting Command Output ===========================
2020-01-20T16:49:30.1993494Z [command]/bin/bash --noprofile --norc /home/vsts/work/_temp/17ec8455-c20a-47b9-84e8-d9a6d4daa511.sh
2020-01-20T16:49:30.1993526Z 
2020-01-20T16:49:30.1996664Z ##[section]Finishing: Disable git automatic line ending conversion
2020-01-20T16:49:30.2003445Z ##[section]Starting: Checkout rust-lang/rust@refs/pull/68298/merge to s
2020-01-20T16:49:30.2005042Z Task         : Get sources
2020-01-20T16:49:30.2005076Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
2020-01-20T16:49:30.2005129Z Version      : 1.0.0
2020-01-20T16:49:30.2005162Z Author       : Microsoft
---
2020-01-20T16:49:31.2054080Z ##[command]git remote add origin https://github.com/rust-lang/rust
2020-01-20T16:49:31.2068847Z ##[command]git config gc.auto 0
2020-01-20T16:49:31.2071615Z ##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
2020-01-20T16:49:31.2074152Z ##[command]git config --get-all http.proxy
2020-01-20T16:49:31.2080515Z ##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/68298/merge:refs/remotes/pull/68298/merge
---
2020-01-20T17:47:39.3629697Z .................................................................................................... 1700/9539
2020-01-20T17:47:46.2919345Z .................................................................................................... 1800/9539
2020-01-20T17:47:58.6672153Z ...................i................................................................................ 1900/9539
2020-01-20T17:48:06.3086912Z .................................................................................................... 2000/9539
2020-01-20T17:48:22.9012153Z .........iiiii...................................................................................... 2100/9539
2020-01-20T17:48:33.2168880Z .................................................................................................... 2300/9539
2020-01-20T17:48:35.8642729Z .................................................................................................... 2400/9539
2020-01-20T17:48:41.7502891Z .................................................................................................... 2500/9539
2020-01-20T17:49:04.0229832Z .................................................................................................... 2600/9539
---
2020-01-20T17:51:56.7280131Z .....................................................i...............i.............................. 4900/9539
2020-01-20T17:52:05.3354100Z .................................................................................................... 5000/9539
2020-01-20T17:52:13.7250642Z ................................................................................................i... 5100/9539
2020-01-20T17:52:19.2960903Z .................................................................................................... 5200/9539
2020-01-20T17:52:30.9253192Z ....................................................................ii.ii...........i............... 5300/9539
2020-01-20T17:52:40.8263943Z .....i.............................................................................................. 5500/9539
2020-01-20T17:52:51.7539277Z .................................................................................................... 5600/9539
2020-01-20T17:52:58.9224084Z ......................................................i............................................. 5700/9539
2020-01-20T17:53:06.3919337Z .................................................................................................... 5800/9539
2020-01-20T17:53:06.3919337Z .................................................................................................... 5800/9539
2020-01-20T17:53:17.1352416Z .................................................................................................... 5900/9539
2020-01-20T17:53:24.5598795Z .............................................ii...i..ii...........i................................. 6000/9539
2020-01-20T17:53:48.0955969Z .................................................................................................... 6200/9539
2020-01-20T17:53:56.9052115Z .................................................................................................... 6300/9539
2020-01-20T17:53:56.9052115Z .................................................................................................... 6300/9539
2020-01-20T17:54:07.5787142Z .........................................................................i..ii...................... 6400/9539
2020-01-20T17:54:45.2476760Z .................................................................................................... 6600/9539
2020-01-20T17:54:48.7488418Z .................................................i.................................................. 6700/9539
2020-01-20T17:54:51.2067528Z .................................................................................................... 6800/9539
2020-01-20T17:54:53.7072178Z ................................................i................................................... 6900/9539
---
2020-01-20T17:56:39.0348848Z .................................................................................................... 7500/9539
2020-01-20T17:56:43.8967175Z .................................................................................................... 7600/9539
2020-01-20T17:56:49.8868918Z .................................................................................................... 7700/9539
2020-01-20T17:56:56.9250113Z .................................................................................................... 7800/9539
2020-01-20T17:57:08.5319569Z .....................................F.............................................................i 7900/9539
2020-01-20T17:57:15.0920757Z iiiiii.............................................................................................. 8000/9539
2020-01-20T17:57:31.4822723Z .................................................................................................... 8200/9539
2020-01-20T17:57:44.0021135Z .................................................................................................... 8300/9539
2020-01-20T17:57:57.6352532Z .................................................................................................... 8400/9539
2020-01-20T17:58:03.9701045Z .................................................................................................... 8500/9539
---
2020-01-20T18:00:09.8041662Z ---- [ui] ui/rmeta-rpass.rs stdout ----
2020-01-20T18:00:09.8041928Z 
2020-01-20T18:00:09.8042414Z error: test compilation failed although it shouldn't!
2020-01-20T18:00:09.8042637Z status: exit code: 1
2020-01-20T18:00:09.8043568Z command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/rmeta-rpass.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-Zdeduplicate-diagnostics=no" "-C" "prefer-dynamic" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/rmeta-rpass/a" "-Crpath" "-O" "-Cdebuginfo=0" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/rmeta-rpass/auxiliary"
2020-01-20T18:00:09.8044233Z ------------------------------------------
2020-01-20T18:00:09.8044443Z 
2020-01-20T18:00:09.8044825Z ------------------------------------------
2020-01-20T18:00:09.8045028Z stderr:
2020-01-20T18:00:09.8045028Z stderr:
2020-01-20T18:00:09.8045692Z ------------------------------------------
2020-01-20T18:00:09.8046294Z error[E0560]: struct `rmeta_aux::Foo` has no field named `field`
2020-01-20T18:00:09.8047011Z    |
2020-01-20T18:00:09.8047176Z LL |     let _ = Foo { field: 42 };
2020-01-20T18:00:09.8047752Z    |                   ^^^^^ help: a field with a similar name exists: `field2`
2020-01-20T18:00:09.8047915Z 
---
2020-01-20T18:00:09.8070427Z thread 'main' panicked at 'Some tests failed', src/tools/compiletest/src/main.rs:387:22
2020-01-20T18:00:09.8070765Z note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
2020-01-20T18:00:09.8081889Z 
2020-01-20T18:00:09.8082179Z 
2020-01-20T18:00:09.8084403Z command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib" "--rustc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "--src-base" "/checkout/src/test/ui" "--build-base" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui" "--stage-id" "stage2-x86_64-unknown-linux-gnu" "--mode" "ui" "--target" "x86_64-unknown-linux-gnu" "--host" "x86_64-unknown-linux-gnu" "--llvm-filecheck" "/usr/lib/llvm-7/bin/FileCheck" "--host-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--target-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--docck-python" "/usr/bin/python2.7" "--lldb-python" "/usr/bin/python2.7" "--gdb" "/usr/bin/gdb" "--quiet" "--llvm-version" "7.0.0\n" "--system-llvm" "--cc" "" "--cxx" "" "--cflags" "" "--llvm-components" "" "--llvm-cxxflags" "" "--adb-path" "adb" "--adb-test-dir" "/data/tmp/work" "--android-cross-path" "" "--color" "always"
2020-01-20T18:00:09.8084966Z 
2020-01-20T18:00:09.8085111Z 
2020-01-20T18:00:09.8093717Z failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
2020-01-20T18:00:09.8093995Z Build completed unsuccessfully in 1:04:52
2020-01-20T18:00:09.8093995Z Build completed unsuccessfully in 1:04:52
2020-01-20T18:00:09.8150437Z == clock drift check ==
2020-01-20T18:00:09.8174117Z   local time: Mon Jan 20 18:00:09 UTC 2020
2020-01-20T18:00:09.9864847Z   network time: Mon, 20 Jan 2020 18:00:09 GMT
2020-01-20T18:00:09.9865672Z == end clock drift check ==
2020-01-20T18:00:10.4389996Z 
2020-01-20T18:00:10.4511257Z ##[error]Bash exited with code '1'.
2020-01-20T18:00:10.4523706Z ##[section]Finishing: Run build
2020-01-20T18:00:10.4546724Z ##[section]Starting: Checkout rust-lang/rust@refs/pull/68298/merge to s
2020-01-20T18:00:10.4548627Z Task         : Get sources
2020-01-20T18:00:10.4548695Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
2020-01-20T18:00:10.4548745Z Version      : 1.0.0
2020-01-20T18:00:10.4548789Z Author       : Microsoft
2020-01-20T18:00:10.4548789Z Author       : Microsoft
2020-01-20T18:00:10.4548855Z Help         : [More Information](https://go.microsoft.com/fwlink/?LinkId=798199)
2020-01-20T18:00:10.4549066Z ==============================================================================
2020-01-20T18:00:10.9058472Z Cleaning any cached credential from repository: rust-lang/rust (GitHub)
2020-01-20T18:00:10.9103157Z ##[section]Finishing: Checkout rust-lang/rust@refs/pull/68298/merge to s
2020-01-20T18:00:10.9229364Z Cleaning up task key
2020-01-20T18:00:10.9230178Z Start cleaning up orphan processes.
2020-01-20T18:00:10.9347926Z Terminate orphan process: pid (3545) (python)
2020-01-20T18:00:10.9641693Z ##[section]Finishing: Finalize Job

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

src/librustc_metadata/locator.rs Outdated Show resolved Hide resolved
src/librustc_metadata/locator.rs Outdated Show resolved Hide resolved
@petrochenkov

This comment has been minimized.

Copy link
Contributor

petrochenkov commented Jan 20, 2020

The failing test case (https://github.com/rust-lang/rust/blob/master/src/test/ui/rmeta-rpass.rs) appears to assert that when we find an rmeta and rlib in the linker path, we should prefer the rlib. I'm not sure why this is true.

From the PR introducing the test it looks like just something to ensure that the behavior doesn't change in unnoticed way.

@Mark-Simulacrum Mark-Simulacrum force-pushed the Mark-Simulacrum:binary-depdep-fix branch from 0f9f382 to 0d0006b Jan 20, 2020
@petrochenkov

This comment has been minimized.

Copy link
Contributor

petrochenkov commented Jan 20, 2020

I consider it in an unstable implementation detail which file is ultimately used if you have incompatible files.

I wanted to write some short overview of this situation and dependencies, perhaps this weekend.

Right now if two versions of the "same library", e.g. "rmeta vs rlib" or "target vs host" in case of "dual proc macros" become unsynchronized, it's pretty much undefined behavior.
Perhaps we can give some guarantees in this case by loading and comparing fingerprints rather than whole metadata, or perhaps not, need to think a bit more.

@petrochenkov

This comment has been minimized.

Copy link
Contributor

petrochenkov commented Jan 20, 2020

Before r+'ing I'd like to get a confirmation from Cargo people that something like

cargo check
modify lib.rs
cargo build

won't produce unsynchronized rmeta (older) and rlib (newer) in the same directory.

It looks like the disambiguators (like 2226b7d67c247d50) in rmeta and rlib file names are different if the source was modified, so the mismatch shouldn't be possible, but I don't know the rules exactly.

@petrochenkov

This comment has been minimized.

Copy link
Contributor

petrochenkov commented Jan 20, 2020

r=me otherwise

When we're producing an rlib, we do not need anything more than an rmeta file
for each of our dependencies (this is indeed utilized by Cargo for pipelining).
Previously, we were still storing the paths of possible rlib/dylib crates, which
meant that they could still plausibly be accessed. With -Zbinary-dep-depinfo,
that meant that Cargo thought that rustc was using both the rlib and an (earlier
emitted) rmeta, and so needed a recompile, as the rlib may have finished writing
*after* compilation started (for more detail, see issue 68149).

This commit changes metadata loading to not store the filepaths of dylib/rlib if
we're going to end up creating an rlib only.
@Mark-Simulacrum Mark-Simulacrum force-pushed the Mark-Simulacrum:binary-depdep-fix branch from 0d0006b to be663bf Jan 20, 2020
@Mark-Simulacrum

This comment has been minimized.

Copy link
Member Author

Mark-Simulacrum commented Jan 20, 2020

Let's r? @alexcrichton then to hopefully resolve the Cargo question (or maybe @ehuss knows better, not sure :)

@alexcrichton

This comment has been minimized.

Copy link
Member

alexcrichton commented Jan 21, 2020

@bors: r=petrochenkov

Code here looks great to me, thanks!

Before r+'ing I'd like to get a confirmation from Cargo people ...

I believe this is correct, due to all the trickiness involved cargo check effectively doesn't share a cache at all with cargo build. We try to keep these all separate.

In any case if I'm remembering incorrectly I suspect we'll figure out quickly!

@bors

This comment has been minimized.

Copy link
Contributor

bors commented Jan 21, 2020

📌 Commit be663bf has been approved by petrochenkov

@Centril

This comment has been minimized.

Copy link
Member

Centril commented Jan 21, 2020

@bors rollup=never to make sure we really figure out quickly.

@bors

This comment has been minimized.

Copy link
Contributor

bors commented Jan 23, 2020

⌛️ Testing commit be663bf with merge 462fc37...

bors added a commit that referenced this pull request Jan 23, 2020
Avoid declaring a fake dependency edge

When we're producing an rlib, we do not need anything more than an rmeta file
for each of our dependencies (this is indeed utilized by Cargo for pipelining).
Previously, we were still storing the paths of possible rlib/dylib crates, which
meant that they could still plausibly be accessed. With -Zbinary-dep-depinfo,
that meant that Cargo thought that rustc was using both the rlib and an (earlier
emitted) rmeta, and so needed a recompile, as the rlib may have finished writing
*after* compilation started (for more detail, see issue 68149).

This commit changes metadata loading to not store the filepaths of dylib/rlib if
we're going to end up creating an rlib only.

Fixes #68149.
@bors

This comment has been minimized.

Copy link
Contributor

bors commented Jan 23, 2020

☀️ Test successful - checks-azure
Approved by: petrochenkov
Pushing 462fc37 to master...

@bors bors added the merged-by-bors label Jan 23, 2020
@bors bors merged commit be663bf into rust-lang:master Jan 23, 2020
5 checks passed
5 checks passed
homu Test successful
Details
pr Build #20200120.46 succeeded
Details
pr (Linux mingw-check) Linux mingw-check succeeded
Details
pr (Linux x86_64-gnu-llvm-7) Linux x86_64-gnu-llvm-7 succeeded
Details
pr (Linux x86_64-gnu-tools) Linux x86_64-gnu-tools succeeded
Details
@Mark-Simulacrum Mark-Simulacrum deleted the Mark-Simulacrum:binary-depdep-fix branch Jan 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

7 participants
You can’t perform that action at this time.