Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.
Sign upWebIDL codegen: Replace cmake with a single Python script #24303
+175
−457
Conversation
|
Tidy and fmt complain |
|
Really great, @SimonSapin! @bors-servo r+ |
|
|
bors-servo
added a commit
that referenced
this pull request
Sep 27, 2019
WebIDL codegen: Replace cmake with a single Python script When [playing around with Cargo’s new timing visualization](https://internals.rust-lang.org/t/exploring-crate-graph-build-times-with-cargo-build-ztimings/10975/21), I was surprised to see the `script` crate’s build script take 76 seconds. I did not expect WebIDL bindings generation to be *that* computationally intensive. It turns out almost all of this time is overhead. The build script uses CMake to generate bindings for each WebIDL file in parallel, but that causes a lot of work to be repeated 366 times: * Starting up a Python VM * Importing (parts of) the Python standard library * Importing ~16k lines of our Python code * Recompiling the latter to bytecode, since we used `python -B` to disable writing `.pyc` files * Deserializing with `cPickle` and recreating in memory the results of parsing all WebIDL files ---- This commit remove the use of CMake and cPickle for the `script` crate. Instead, all WebIDL bindings generation is done sequentially in a single Python process. This takes 2 to 3 seconds. <!-- Reviewable:start --> --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/24303) <!-- Reviewable:end -->
|
|
|
@bors-servo retry |
bors-servo
added a commit
that referenced
this pull request
Sep 27, 2019
WebIDL codegen: Replace cmake with a single Python script When [playing around with Cargo’s new timing visualization](https://internals.rust-lang.org/t/exploring-crate-graph-build-times-with-cargo-build-ztimings/10975/21), I was surprised to see the `script` crate’s build script take 76 seconds. I did not expect WebIDL bindings generation to be *that* computationally intensive. It turns out almost all of this time is overhead. The build script uses CMake to generate bindings for each WebIDL file in parallel, but that causes a lot of work to be repeated 366 times: * Starting up a Python VM * Importing (parts of) the Python standard library * Importing ~16k lines of our Python code * Recompiling the latter to bytecode, since we used `python -B` to disable writing `.pyc` files * Deserializing with `cPickle` and recreating in memory the results of parsing all WebIDL files ---- This commit remove the use of CMake and cPickle for the `script` crate. Instead, all WebIDL bindings generation is done sequentially in a single Python process. This takes 2 to 3 seconds. <!-- Reviewable:start --> --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/24303) <!-- Reviewable:end -->
|
|
When playing around with Cargo’s new timing visualization: https://internals.rust-lang.org/t/exploring-crate-graph-build-times-with-cargo-build-ztimings/10975/21 … I was surprised to see the `script` crate’s build script take 76 seconds. I did not expect WebIDL bindings generation to be *that* computationally intensive. It turns out almost all of this time is overhead. The build script uses CMake to generate bindings for each WebIDL file in parallel, but that causes a lot of work to be repeated 366 times: * Starting up a Python VM * Importing (parts of) the Python standard library * Importing ~16k lines of our Python code * Recompiling the latter to bytecode, since we used `python -B` to disable writing `.pyc` file * Deserializing with `cPickle` and recreating in memory the results of parsing all WebIDL files ---- This commit remove the use of CMake and cPickle for the `script` crate. Instead, all WebIDL bindings generation is done sequentially in a single Python process. This takes 2 to 3 seconds.
|
Fixed a Windows-specific failure: @bors-servo r=nox |
|
|
bors-servo
added a commit
that referenced
this pull request
Sep 27, 2019
WebIDL codegen: Replace cmake with a single Python script When [playing around with Cargo’s new timing visualization](https://internals.rust-lang.org/t/exploring-crate-graph-build-times-with-cargo-build-ztimings/10975/21), I was surprised to see the `script` crate’s build script take 76 seconds. I did not expect WebIDL bindings generation to be *that* computationally intensive. It turns out almost all of this time is overhead. The build script uses CMake to generate bindings for each WebIDL file in parallel, but that causes a lot of work to be repeated 366 times: * Starting up a Python VM * Importing (parts of) the Python standard library * Importing ~16k lines of our Python code * Recompiling the latter to bytecode, since we used `python -B` to disable writing `.pyc` files * Deserializing with `cPickle` and recreating in memory the results of parsing all WebIDL files ---- This commit remove the use of CMake and cPickle for the `script` crate. Instead, all WebIDL bindings generation is done sequentially in a single Python process. This takes 2 to 3 seconds. <!-- Reviewable:start --> --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/24303) <!-- Reviewable:end -->
Are you drunk, @bors-servo? |
|
|
|
@bors-servo retry #20418 #20823 #19221 #16216 |
bors-servo
added a commit
that referenced
this pull request
Sep 29, 2019
WebIDL codegen: Replace cmake with a single Python script When [playing around with Cargo’s new timing visualization](https://internals.rust-lang.org/t/exploring-crate-graph-build-times-with-cargo-build-ztimings/10975/21), I was surprised to see the `script` crate’s build script take 76 seconds. I did not expect WebIDL bindings generation to be *that* computationally intensive. It turns out almost all of this time is overhead. The build script uses CMake to generate bindings for each WebIDL file in parallel, but that causes a lot of work to be repeated 366 times: * Starting up a Python VM * Importing (parts of) the Python standard library * Importing ~16k lines of our Python code * Recompiling the latter to bytecode, since we used `python -B` to disable writing `.pyc` files * Deserializing with `cPickle` and recreating in memory the results of parsing all WebIDL files ---- This commit remove the use of CMake and cPickle for the `script` crate. Instead, all WebIDL bindings generation is done sequentially in a single Python process. This takes 2 to 3 seconds. <!-- Reviewable:start --> --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/24303) <!-- Reviewable:end -->
|
|
|
Failure in |
|
I'm 99.99% sure this happened before, I think @jdm solved it last time. |
|
I’ve filed rust-lang/cargo#7458 to ask around. |
|
@bors-servo r+ |
|
|
bors-servo
added a commit
that referenced
this pull request
Sep 30, 2019
WebIDL codegen: Replace cmake with a single Python script When [playing around with Cargo’s new timing visualization](https://internals.rust-lang.org/t/exploring-crate-graph-build-times-with-cargo-build-ztimings/10975/21), I was surprised to see the `script` crate’s build script take 76 seconds. I did not expect WebIDL bindings generation to be *that* computationally intensive. It turns out almost all of this time is overhead. The build script uses CMake to generate bindings for each WebIDL file in parallel, but that causes a lot of work to be repeated 366 times: * Starting up a Python VM * Importing (parts of) the Python standard library * Importing ~16k lines of our Python code * Recompiling the latter to bytecode, since we used `python -B` to disable writing `.pyc` files * Deserializing with `cPickle` and recreating in memory the results of parsing all WebIDL files ---- This commit remove the use of CMake and cPickle for the `script` crate. Instead, all WebIDL bindings generation is done sequentially in a single Python process. This takes 2 to 3 seconds. <!-- Reviewable:start --> --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/24303) <!-- Reviewable:end -->
|
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
SimonSapin commentedSep 27, 2019
•
edited
When playing around with Cargo’s new timing visualization, I was surprised to see the
scriptcrate’s build script take 76 seconds. I did not expect WebIDL bindings generation to be that computationally intensive.It turns out almost all of this time is overhead. The build script uses CMake to generate bindings for each WebIDL file in parallel, but that causes a lot of work to be repeated 366 times:
python -Bto disable writing.pycfilescPickleand recreating in memory the results of parsing all WebIDL filesThis commit remove the use of CMake and cPickle for the
scriptcrate. Instead, all WebIDL bindings generation is done sequentially in a single Python process. This takes 2 to 3 seconds.This change is