Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

save-analysis: Use serde instead of libserialize to dump JSON data #60053

Merged
merged 3 commits into from
Apr 22, 2019

Conversation

Xanewok
Copy link
Member

@Xanewok Xanewok commented Apr 17, 2019

This breaks the save-analysis infrastructure (which also includes rls-{analysis, data, span} crates) from depending on rustc_serialize and so we can start moving them to being supported on stable without implementing Decodable et al. by hand for data structures defined there.

Notable benefits:

  • we drop the awkward raw byte PathBuf serialization (until now (de)serialized as &[u8])
  • faster (hopefully noticeable for inner crate dependencies for the RLS workloads)
  • we can easily explore the binary serialization backend (which we planned to do for save-analysis anyway)

This should be merged together with an update to RLS (rust-lang/rls#1435), which technically could be included right now because we can use the bundled rls-analysis here directly, however I'd prefer to publish this to crates.io first (rust-lang/rls#1434, cc @nrc) and use the published version, instead.
Includes rust-lang/rls#1436.

@matklad @nikomatsakis This is also important for the potential RLS 1.0 - 2.0 bridge we talked about on Zulip today

@rust-highfive
Copy link
Collaborator

r? @varkor

(rust_highfive has picked a reviewer for you, use r? to override)

@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Apr 17, 2019
@Xanewok
Copy link
Member Author

Xanewok commented Apr 17, 2019

I tested the change by compiling rustc and using it to compile RLS, which seemed to work normally and also did spit the data using the new serde backend.

FWIW I also tested running RLS on a crate which pulled in serde, serde_derive and friends and nothing seemed to break, so here's hoping everything works as expected 🤞

Copy link
Member

@matklad matklad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

src/librustc_save_analysis/json_dumper.rs Outdated Show resolved Hide resolved
@Xanewok Xanewok marked this pull request as ready for review April 18, 2019 09:42
@Xanewok Xanewok changed the title [WIP] save-analysis: Use serde instead of libserialize to dump JSON data save-analysis: Use serde instead of libserialize to dump JSON data Apr 18, 2019
@Xanewok Xanewok closed this Apr 18, 2019
@Xanewok Xanewok reopened this Apr 18, 2019
@nrc
Copy link
Member

nrc commented Apr 19, 2019

@bors: r+

@bors
Copy link
Contributor

bors commented Apr 19, 2019

📌 Commit 8498ba204b4c83c21f29d4918057a96cd601baef has been approved by nrc

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 19, 2019
@Xanewok
Copy link
Member Author

Xanewok commented Apr 19, 2019

Added a test, which checks that reading a malformed config doesn't panic

@bors r=nrc

@bors
Copy link
Contributor

bors commented Apr 19, 2019

📌 Commit b03123426c60dc22d84291a0fcecef67451514a8 has been approved by nrc

@bors
Copy link
Contributor

bors commented Apr 19, 2019

🔒 Merge conflict

This pull request and the master branch diverged in a way that cannot be automatically merged. Please rebase on top of the latest master branch, and let the reviewer approve again.

How do I rebase?

Assuming self is your fork and upstream is this repository, you can resolve the conflict following these steps:

  1. git checkout serde-save-analysis (switch to your branch)
  2. git fetch upstream master (retrieve the latest master)
  3. git rebase upstream/master -p (rebase on top of it)
  4. Follow the on-screen instruction to resolve conflicts (check git status if you got lost).
  5. git push self serde-save-analysis --force-with-lease (update this PR)

You may also read Git Rebasing to Resolve Conflicts by Drew Blessing for a short tutorial.

Please avoid the "Resolve conflicts" button on GitHub. It uses git merge instead of git rebase which makes the PR commit history more difficult to read.

Sometimes step 4 will complete without asking for resolution. This is usually due to difference between how Cargo.lock conflict is handled during merge and rebase. This is normal, and you should still perform step 5 to update this PR.

Error message
warning: Cannot merge binary files: Cargo.lock (HEAD vs. heads/homu-tmp)
Auto-merging Cargo.lock
CONFLICT (content): Merge conflict in Cargo.lock
Automatic merge failed; fix conflicts and then commit the result.

@bors bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Apr 19, 2019
@Xanewok
Copy link
Member Author

Xanewok commented Apr 19, 2019

That's weird... rebased just fine.

@bors r=nrc

@bors
Copy link
Contributor

bors commented Apr 19, 2019

📌 Commit bc8af9a has been approved by nrc

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Apr 19, 2019
@bors
Copy link
Contributor

bors commented Apr 19, 2019

⌛ Testing commit bc8af9a with merge 227d009d96ba8703729c5ae049ddf0aec20041f9...

@bors
Copy link
Contributor

bors commented Apr 19, 2019

💔 Test failed - checks-travis

@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-llvm-6.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_time:end:1382a100:start=1555848997165473943,finish=1555848997939887501,duration=774413558
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
$ export GCP_CACHE_BUCKET=rust-lang-ci-cache
$ export AWS_ACCESS_KEY_ID=AKIA46X5W6CZEJZ6XT55
---
travis_time:start:test_assembly
Check compiletest suite=assembly mode=assembly (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:13:33] 
[01:13:33] running 9 tests
[01:13:33] iiiiiiiii
[01:13:33] 
[01:13:33]  finished in 0.149
[01:13:33] travis_fold:end:test_assembly

---
travis_time:start:test_debuginfo
Check compiletest suite=debuginfo mode=debuginfo-both (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:13:49] 
[01:13:49] running 121 tests
[01:14:13] .iiiii...i.....i..i...i..i.i.i..i.ii...i.....i..i....i..........iiii..........i...ii...i.......ii.i. 100/121
[01:14:18] i.i......iii.i.....ii
[01:14:18] 
[01:14:18]  finished in 29.217
[01:14:18] travis_fold:end:test_debuginfo

---
travis_time:start:test_run-pass-fulldeps
Check compiletest suite=run-pass-fulldeps mode=run-pass (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:14:25] 
[01:14:25] running 48 tests
[01:15:54] ................................F..............test [run-pass] run-pass-fulldeps/myriad-closures.rs has been running for over 60 seconds
[01:18:06] note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
[01:18:06] .
[01:18:06] failures:
[01:18:06] 
[01:18:06] 
[01:18:06] ---- [run-pass] run-pass-fulldeps/newtype_index.rs stdout ----
[01:18:06] 
[01:18:06] error: test compilation failed although it shouldn't!
[01:18:06] status: exit code: 1
[01:18:06] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/run-pass-fulldeps/newtype_index.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-C" "prefer-dynamic" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-pass-fulldeps/newtype_index/a" "-Crpath" "-O" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-pass-fulldeps/newtype_index/auxiliary"
[01:18:06] ------------------------------------------
[01:18:06] 
[01:18:06] ------------------------------------------
[01:18:06] stderr:
---
[01:18:06] test result: FAILED. 47 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out
[01:18:06] 
[01:18:06] 
[01:18:06] 
[01:18:06] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib" "--rustc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "--src-base" "/checkout/src/test/run-pass-fulldeps" "--build-base" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-pass-fulldeps" "--stage-id" "stage2-x86_64-unknown-linux-gnu" "--mode" "run-pass" "--target" "x86_64-unknown-linux-gnu" "--host" "x86_64-unknown-linux-gnu" "--llvm-filecheck" "/usr/lib/llvm-6.0/bin/FileCheck" "--host-rustcflags" "-Crpath -O -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--target-rustcflags" "-Crpath -O -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--docck-python" "/usr/bin/python2.7" "--lldb-python" "/usr/bin/python2.7" "--gdb" "/usr/bin/gdb" "--quiet" "--llvm-version" "6.0.0\n" "--system-llvm" "--cc" "" "--cxx" "" "--cflags" "" "--llvm-components" "" "--llvm-cxxflags" "" "--adb-path" "adb" "--adb-test-dir" "/data/tmp/work" "--android-cross-path" "" "--color" "always"
[01:18:06] 
[01:18:06] 
[01:18:06] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
[01:18:06] Build completed unsuccessfully in 0:16:06
[01:18:06] Build completed unsuccessfully in 0:16:06
[01:18:06] Makefile:48: recipe for target 'check' failed
[01:18:06] make: *** [check] Error 1
The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:057758e0
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
Sun Apr 21 13:34:54 UTC 2019

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@Xanewok
Copy link
Member Author

Xanewok commented Apr 21, 2019

Updated rls-* crates to explicitly not use serde_derive (but rather manual proc macro expansion, which... is still suboptimal).

@bors r=nrc

@bors
Copy link
Contributor

bors commented Apr 21, 2019

📌 Commit 2dccaa7 has been approved by nrc

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 21, 2019
@bors
Copy link
Contributor

bors commented Apr 21, 2019

⌛ Testing commit 2dccaa7 with merge f51fc22e2711493012c954049be8c4c35c545968...

@bors
Copy link
Contributor

bors commented Apr 21, 2019

💔 Test failed - checks-travis

@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-tools of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
[01:30:37] Verifying status of edition-guide...
[01:30:37] Verifying status of rls...
[01:30:37] This PR updated 'src/tools/rls', verifying if status is 'test-pass'...
[01:30:37] 
[01:30:37] ⚠️ We detected that this PR updated 'rls', but its tests failed.
[01:30:37] 
[01:30:37] If you do intend to update 'rls', please check the error messages above and
[01:30:37] commit another update.
[01:30:37] 
[01:30:37] If you do NOT intend to update 'rls', please ensure you did not accidentally
[01:30:37] change the submodule at 'src/tools/rls'. You may ask your reviewer for the
[01:30:37] proper steps.
The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 3.
travis_time:start:1ddd8af1
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
Sun Apr 21 18:13:28 UTC 2019
---
travis_time:end:009ee14c:start=1555870410253555794,finish=1555870410269812185,duration=16256391
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:21cb2f48
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:1f98c480
travis_time:start:1f98c480
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:11cd13cc
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@bors bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Apr 21, 2019
@Xanewok
Copy link
Member Author

Xanewok commented Apr 21, 2019

Argh, RLS broke again due to Clippy, will need to fix that first

@Xanewok
Copy link
Member Author

Xanewok commented Apr 22, 2019

Looks like #60158 is almost merged, with

[01:57:55] The state of "clippy-driver" has changed from "build-fail" to "test-pass"
[01:57:55] The state of "rls" has changed from "build-fail" to "test-pass"

and the queue has only one PR (59114) pending without Cargo.lock, so I'll queue this up for the night:
@bors retry

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 22, 2019
@bors
Copy link
Contributor

bors commented Apr 22, 2019

⌛ Testing commit 2dccaa7 with merge 8f06188...

bors added a commit that referenced this pull request Apr 22, 2019
save-analysis: Use serde instead of libserialize to dump JSON data

This breaks the save-analysis infrastructure (which also includes `rls-{analysis, data, span}` crates) from depending on rustc_serialize and so we can start moving them to being supported on stable without implementing `Decodable` et al. by hand for data structures defined there.

Notable benefits:
- we drop the awkward raw byte `PathBuf` [serialization](https://gist.github.com/Xanewok/f4fe8564d0dc0c3ab1dbc244279ff895) (until now (de)serialized as `&[u8]`)
- [faster](https://github.com/serde-rs/json-benchmark) (hopefully noticeable for inner crate dependencies for the RLS workloads)
- we can easily explore the binary serialization backend (which we planned to do for save-analysis anyway)

~This should be merged together with an update to RLS (rust-lang/rls#1435), which technically could be included right now because we can use the bundled `rls-analysis` here directly, however I'd prefer to publish this to crates.io first (rust-lang/rls#1434, cc @nrc) and use the published version, instead.~
Includes rust-lang/rls#1436.

@matklad @nikomatsakis This is also important for the potential RLS 1.0 - 2.0 bridge we talked about on Zulip today
@bors
Copy link
Contributor

bors commented Apr 22, 2019

☀️ Test successful - checks-travis, status-appveyor
Approved by: nrc
Pushing 8f06188 to master...

@bors bors added the merged-by-bors This PR was explicitly merged by bors. label Apr 22, 2019
@bors bors merged commit 2dccaa7 into rust-lang:master Apr 22, 2019
@rust-highfive
Copy link
Collaborator

📣 Toolstate changed by #60053!

Tested on commit 8f06188.
Direct link to PR: #60053

💔 clippy-driver on windows: test-pass → test-fail (cc @Manishearth @llogiq @mcarton @oli-obk @phansch, @rust-lang/infra).
💔 clippy-driver on linux: test-pass → test-fail (cc @Manishearth @llogiq @mcarton @oli-obk @phansch, @rust-lang/infra).

rust-highfive added a commit to rust-lang-nursery/rust-toolstate that referenced this pull request Apr 22, 2019
Tested on commit rust-lang/rust@8f06188.
Direct link to PR: <rust-lang/rust#60053>

💔 clippy-driver on windows: test-pass → test-fail (cc @Manishearth @llogiq @mcarton @oli-obk @phansch, @rust-lang/infra).
💔 clippy-driver on linux: test-pass → test-fail (cc @Manishearth @llogiq @mcarton @oli-obk @phansch, @rust-lang/infra).
@Xanewok Xanewok deleted the serde-save-analysis branch April 23, 2019 08:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
merged-by-bors This PR was explicitly merged by bors. S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants