Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

save-analysis: Use serde instead of libserialize to dump JSON data #60053

Merged
merged 3 commits into from Apr 22, 2019

Conversation

Projects
None yet
6 participants
@Xanewok
Copy link
Member

commented Apr 17, 2019

This breaks the save-analysis infrastructure (which also includes rls-{analysis, data, span} crates) from depending on rustc_serialize and so we can start moving them to being supported on stable without implementing Decodable et al. by hand for data structures defined there.

Notable benefits:

  • we drop the awkward raw byte PathBuf serialization (until now (de)serialized as &[u8])
  • faster (hopefully noticeable for inner crate dependencies for the RLS workloads)
  • we can easily explore the binary serialization backend (which we planned to do for save-analysis anyway)

This should be merged together with an update to RLS (rust-lang/rls#1435), which technically could be included right now because we can use the bundled rls-analysis here directly, however I'd prefer to publish this to crates.io first (rust-lang/rls#1434, cc @nrc) and use the published version, instead.
Includes rust-lang/rls#1436.

@matklad @nikomatsakis This is also important for the potential RLS 1.0 - 2.0 bridge we talked about on Zulip today

@rust-highfive

This comment has been minimized.

Copy link
Collaborator

commented Apr 17, 2019

r? @varkor

(rust_highfive has picked a reviewer for you, use r? to override)

@Xanewok

This comment has been minimized.

Copy link
Member Author

commented Apr 17, 2019

I tested the change by compiling rustc and using it to compile RLS, which seemed to work normally and also did spit the data using the new serde backend.

FWIW I also tested running RLS on a crate which pulled in serde, serde_derive and friends and nothing seemed to break, so here's hoping everything works as expected 🤞

@matklad
Copy link
Member

left a comment

👍

Show resolved Hide resolved src/librustc_save_analysis/json_dumper.rs Outdated

@Xanewok Xanewok marked this pull request as ready for review Apr 18, 2019

@Xanewok Xanewok changed the title [WIP] save-analysis: Use serde instead of libserialize to dump JSON data save-analysis: Use serde instead of libserialize to dump JSON data Apr 18, 2019

@Xanewok Xanewok closed this Apr 18, 2019

@Xanewok Xanewok reopened this Apr 18, 2019

@nrc

nrc approved these changes Apr 19, 2019

@nrc

This comment has been minimized.

Copy link
Member

commented Apr 19, 2019

@bors: r+

@bors

This comment has been minimized.

Copy link
Contributor

commented Apr 19, 2019

📌 Commit 8498ba2 has been approved by nrc

@Xanewok

This comment has been minimized.

Copy link
Member Author

commented Apr 19, 2019

Added a test, which checks that reading a malformed config doesn't panic

@bors r=nrc

@bors

This comment has been minimized.

Copy link
Contributor

commented Apr 19, 2019

📌 Commit b031234 has been approved by nrc

@bors

This comment has been minimized.

Copy link
Contributor

commented Apr 19, 2019

🔒 Merge conflict

This pull request and the master branch diverged in a way that cannot be automatically merged. Please rebase on top of the latest master branch, and let the reviewer approve again.

How do I rebase?

Assuming self is your fork and upstream is this repository, you can resolve the conflict following these steps:

  1. git checkout serde-save-analysis (switch to your branch)
  2. git fetch upstream master (retrieve the latest master)
  3. git rebase upstream/master -p (rebase on top of it)
  4. Follow the on-screen instruction to resolve conflicts (check git status if you got lost).
  5. git push self serde-save-analysis --force-with-lease (update this PR)

You may also read Git Rebasing to Resolve Conflicts by Drew Blessing for a short tutorial.

Please avoid the "Resolve conflicts" button on GitHub. It uses git merge instead of git rebase which makes the PR commit history more difficult to read.

Sometimes step 4 will complete without asking for resolution. This is usually due to difference between how Cargo.lock conflict is handled during merge and rebase. This is normal, and you should still perform step 5 to update this PR.

Error message
warning: Cannot merge binary files: Cargo.lock (HEAD vs. heads/homu-tmp)
Auto-merging Cargo.lock
CONFLICT (content): Merge conflict in Cargo.lock
Automatic merge failed; fix conflicts and then commit the result.

@Xanewok Xanewok force-pushed the Xanewok:serde-save-analysis branch from b031234 to bc8af9a Apr 19, 2019

@Xanewok

This comment has been minimized.

Copy link
Member Author

commented Apr 19, 2019

That's weird... rebased just fine.

@bors r=nrc

@bors

This comment has been minimized.

Copy link
Contributor

commented Apr 19, 2019

📌 Commit bc8af9a has been approved by nrc

@bors

This comment has been minimized.

Copy link
Contributor

commented Apr 19, 2019

⌛️ Testing commit bc8af9a with merge 227d009...

bors added a commit that referenced this pull request Apr 19, 2019

Auto merge of #60053 - Xanewok:serde-save-analysis, r=nrc
save-analysis: Use serde instead of libserialize to dump JSON data

This breaks the save-analysis infrastructure (which also includes `rls-{analysis, data, span}` crates) from depending on rustc_serialize and so we can start moving them to being supported on stable without implementing `Decodable` et al. by hand for data structures defined there.

Notable benefits:
- we drop the awkward raw byte `PathBuf` [serialization](https://gist.github.com/Xanewok/f4fe8564d0dc0c3ab1dbc244279ff895) (until now (de)serialized as `&[u8]`)
- [faster](https://github.com/serde-rs/json-benchmark) (hopefully noticeable for inner crate dependencies for the RLS workloads)
- we can easily explore the binary serialization backend (which we planned to do for save-analysis anyway)

~This should be merged together with an update to RLS (rust-lang/rls#1435), which technically could be included right now because we can use the bundled `rls-analysis` here directly, however I'd prefer to publish this to crates.io first (rust-lang/rls#1434, cc @nrc) and use the published version, instead.~
Includes rust-lang/rls#1436.

@matklad @nikomatsakis This is also important for the potential RLS 1.0 - 2.0 bridge we talked about on Zulip today
@bors

This comment has been minimized.

Copy link
Contributor

commented Apr 19, 2019

💔 Test failed - checks-travis

Switch to serde-enabled rls-* and update RLS appropriately
This also bumps RLS version to 1.36.
The updated rls-* packages use serde but *not* serde_derive thanks to
manual proc macro expansion. This is a hack, since rustc cannot handle
crates.io proc macros (duplicated in tools) when cross-compiling, so
that's the best we can do in order to support serde_json in save-analysis.

@Xanewok Xanewok force-pushed the Xanewok:serde-save-analysis branch from bc8af9a to 4fb570d Apr 21, 2019

@rust-highfive

This comment has been minimized.

Copy link
Collaborator

commented Apr 21, 2019

The job x86_64-gnu-llvm-6.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_time:end:1382a100:start=1555848997165473943,finish=1555848997939887501,duration=774413558
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
$ export GCP_CACHE_BUCKET=rust-lang-ci-cache
$ export AWS_ACCESS_KEY_ID=AKIA46X5W6CZEJZ6XT55
---
travis_time:start:test_assembly
Check compiletest suite=assembly mode=assembly (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:13:33] 
[01:13:33] running 9 tests
[01:13:33] iiiiiiiii
[01:13:33] 
[01:13:33]  finished in 0.149
[01:13:33] travis_fold:end:test_assembly

---
travis_time:start:test_debuginfo
Check compiletest suite=debuginfo mode=debuginfo-both (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:13:49] 
[01:13:49] running 121 tests
[01:14:13] .iiiii...i.....i..i...i..i.i.i..i.ii...i.....i..i....i..........iiii..........i...ii...i.......ii.i. 100/121
[01:14:18] i.i......iii.i.....ii
[01:14:18] 
[01:14:18]  finished in 29.217
[01:14:18] travis_fold:end:test_debuginfo

---
travis_time:start:test_run-pass-fulldeps
Check compiletest suite=run-pass-fulldeps mode=run-pass (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:14:25] 
[01:14:25] running 48 tests
[01:15:54] ................................F..............test [run-pass] run-pass-fulldeps/myriad-closures.rs has been running for over 60 seconds
[01:18:06] note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
[01:18:06] .
[01:18:06] failures:
[01:18:06] 
[01:18:06] 
[01:18:06] ---- [run-pass] run-pass-fulldeps/newtype_index.rs stdout ----
[01:18:06] 
[01:18:06] error: test compilation failed although it shouldn't!
[01:18:06] status: exit code: 1
[01:18:06] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/run-pass-fulldeps/newtype_index.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-C" "prefer-dynamic" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-pass-fulldeps/newtype_index/a" "-Crpath" "-O" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-pass-fulldeps/newtype_index/auxiliary"
[01:18:06] ------------------------------------------
[01:18:06] 
[01:18:06] ------------------------------------------
[01:18:06] stderr:
---
[01:18:06] test result: FAILED. 47 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out
[01:18:06] 
[01:18:06] 
[01:18:06] 
[01:18:06] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib" "--rustc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "--src-base" "/checkout/src/test/run-pass-fulldeps" "--build-base" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-pass-fulldeps" "--stage-id" "stage2-x86_64-unknown-linux-gnu" "--mode" "run-pass" "--target" "x86_64-unknown-linux-gnu" "--host" "x86_64-unknown-linux-gnu" "--llvm-filecheck" "/usr/lib/llvm-6.0/bin/FileCheck" "--host-rustcflags" "-Crpath -O -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--target-rustcflags" "-Crpath -O -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--docck-python" "/usr/bin/python2.7" "--lldb-python" "/usr/bin/python2.7" "--gdb" "/usr/bin/gdb" "--quiet" "--llvm-version" "6.0.0\n" "--system-llvm" "--cc" "" "--cxx" "" "--cflags" "" "--llvm-components" "" "--llvm-cxxflags" "" "--adb-path" "adb" "--adb-test-dir" "/data/tmp/work" "--android-cross-path" "" "--color" "always"
[01:18:06] 
[01:18:06] 
[01:18:06] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
[01:18:06] Build completed unsuccessfully in 0:16:06
[01:18:06] Build completed unsuccessfully in 0:16:06
[01:18:06] Makefile:48: recipe for target 'check' failed
[01:18:06] make: *** [check] Error 1
The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:057758e0
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
Sun Apr 21 13:34:54 UTC 2019

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@Xanewok

This comment has been minimized.

Copy link
Member Author

commented Apr 21, 2019

Updated rls-* crates to explicitly not use serde_derive (but rather manual proc macro expansion, which... is still suboptimal).

@bors r=nrc

@bors

This comment has been minimized.

Copy link
Contributor

commented Apr 21, 2019

📌 Commit 2dccaa7 has been approved by nrc

bors added a commit that referenced this pull request Apr 21, 2019

Auto merge of #60053 - Xanewok:serde-save-analysis, r=nrc
save-analysis: Use serde instead of libserialize to dump JSON data

This breaks the save-analysis infrastructure (which also includes `rls-{analysis, data, span}` crates) from depending on rustc_serialize and so we can start moving them to being supported on stable without implementing `Decodable` et al. by hand for data structures defined there.

Notable benefits:
- we drop the awkward raw byte `PathBuf` [serialization](https://gist.github.com/Xanewok/f4fe8564d0dc0c3ab1dbc244279ff895) (until now (de)serialized as `&[u8]`)
- [faster](https://github.com/serde-rs/json-benchmark) (hopefully noticeable for inner crate dependencies for the RLS workloads)
- we can easily explore the binary serialization backend (which we planned to do for save-analysis anyway)

~This should be merged together with an update to RLS (rust-lang/rls#1435), which technically could be included right now because we can use the bundled `rls-analysis` here directly, however I'd prefer to publish this to crates.io first (rust-lang/rls#1434, cc @nrc) and use the published version, instead.~
Includes rust-lang/rls#1436.

@matklad @nikomatsakis This is also important for the potential RLS 1.0 - 2.0 bridge we talked about on Zulip today
@bors

This comment has been minimized.

Copy link
Contributor

commented Apr 21, 2019

⌛️ Testing commit 2dccaa7 with merge f51fc22...

@bors

This comment has been minimized.

Copy link
Contributor

commented Apr 21, 2019

💔 Test failed - checks-travis

@rust-highfive

This comment has been minimized.

Copy link
Collaborator

commented Apr 21, 2019

The job x86_64-gnu-tools of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
[01:30:37] Verifying status of edition-guide...
[01:30:37] Verifying status of rls...
[01:30:37] This PR updated 'src/tools/rls', verifying if status is 'test-pass'...
[01:30:37] 
[01:30:37] ⚠️ We detected that this PR updated 'rls', but its tests failed.
[01:30:37] 
[01:30:37] If you do intend to update 'rls', please check the error messages above and
[01:30:37] commit another update.
[01:30:37] 
[01:30:37] If you do NOT intend to update 'rls', please ensure you did not accidentally
[01:30:37] change the submodule at 'src/tools/rls'. You may ask your reviewer for the
[01:30:37] proper steps.
The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 3.
travis_time:start:1ddd8af1
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
Sun Apr 21 18:13:28 UTC 2019
---
travis_time:end:009ee14c:start=1555870410253555794,finish=1555870410269812185,duration=16256391
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:21cb2f48
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:1f98c480
travis_time:start:1f98c480
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:11cd13cc
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@Xanewok

This comment has been minimized.

Copy link
Member Author

commented Apr 21, 2019

Argh, RLS broke again due to Clippy, will need to fix that first

@Xanewok

This comment has been minimized.

Copy link
Member Author

commented Apr 22, 2019

Looks like #60158 is almost merged, with

[01:57:55] The state of "clippy-driver" has changed from "build-fail" to "test-pass"
[01:57:55] The state of "rls" has changed from "build-fail" to "test-pass"

and the queue has only one PR (59114) pending without Cargo.lock, so I'll queue this up for the night:
@bors retry

@bors

This comment has been minimized.

Copy link
Contributor

commented Apr 22, 2019

⌛️ Testing commit 2dccaa7 with merge 8f06188...

bors added a commit that referenced this pull request Apr 22, 2019

Auto merge of #60053 - Xanewok:serde-save-analysis, r=nrc
save-analysis: Use serde instead of libserialize to dump JSON data

This breaks the save-analysis infrastructure (which also includes `rls-{analysis, data, span}` crates) from depending on rustc_serialize and so we can start moving them to being supported on stable without implementing `Decodable` et al. by hand for data structures defined there.

Notable benefits:
- we drop the awkward raw byte `PathBuf` [serialization](https://gist.github.com/Xanewok/f4fe8564d0dc0c3ab1dbc244279ff895) (until now (de)serialized as `&[u8]`)
- [faster](https://github.com/serde-rs/json-benchmark) (hopefully noticeable for inner crate dependencies for the RLS workloads)
- we can easily explore the binary serialization backend (which we planned to do for save-analysis anyway)

~This should be merged together with an update to RLS (rust-lang/rls#1435), which technically could be included right now because we can use the bundled `rls-analysis` here directly, however I'd prefer to publish this to crates.io first (rust-lang/rls#1434, cc @nrc) and use the published version, instead.~
Includes rust-lang/rls#1436.

@matklad @nikomatsakis This is also important for the potential RLS 1.0 - 2.0 bridge we talked about on Zulip today
@bors

This comment has been minimized.

Copy link
Contributor

commented Apr 22, 2019

☀️ Test successful - checks-travis, status-appveyor
Approved by: nrc
Pushing 8f06188 to master...

@bors bors added the merged-by-bors label Apr 22, 2019

@bors bors merged commit 2dccaa7 into rust-lang:master Apr 22, 2019

2 checks passed

Travis CI - Pull Request Build Passed
Details
homu Test successful
Details
@rust-highfive

This comment has been minimized.

Copy link
Collaborator

commented Apr 22, 2019

📣 Toolstate changed by #60053!

Tested on commit 8f06188.
Direct link to PR: #60053

💔 clippy-driver on windows: test-pass → test-fail (cc @Manishearth @llogiq @mcarton @oli-obk @phansch, @rust-lang/infra).
💔 clippy-driver on linux: test-pass → test-fail (cc @Manishearth @llogiq @mcarton @oli-obk @phansch, @rust-lang/infra).

rust-highfive added a commit to rust-lang-nursery/rust-toolstate that referenced this pull request Apr 22, 2019

📣 Toolstate changed by rust-lang/rust#60053!
Tested on commit rust-lang/rust@8f06188.
Direct link to PR: <rust-lang/rust#60053>

💔 clippy-driver on windows: test-pass → test-fail (cc @Manishearth @llogiq @mcarton @oli-obk @phansch, @rust-lang/infra).
💔 clippy-driver on linux: test-pass → test-fail (cc @Manishearth @llogiq @mcarton @oli-obk @phansch, @rust-lang/infra).

@Xanewok Xanewok deleted the Xanewok:serde-save-analysis branch Apr 23, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.