Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Merged by Bors] - Update the PR benchmarks action #1774

Closed
wants to merge 6 commits into from

Conversation

Razican
Copy link
Member

@Razican Razican commented Jan 3, 2022

This is just for debugging for now. If we find out a fix, we can merge it.

@Razican Razican added the benchmark Issues and PRs related to the benchmark subsystem. label Jan 3, 2022
@github-actions
Copy link

github-actions bot commented Jan 3, 2022

Test262 conformance changes

VM implementation

Test result main count PR count difference
Total 87,200 87,200 0
Passed 40,828 40,828 0
Ignored 19,493 19,493 0
Failed 26,879 26,879 0
Panics 0 0 0
Conformance 46.82% 46.82% 0.00%

@codecov
Copy link

codecov bot commented Jan 3, 2022

Codecov Report

Merging #1774 (e69d80c) into main (d831ff3) will decrease coverage by 0.78%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #1774      +/-   ##
==========================================
- Coverage   56.30%   55.51%   -0.79%     
==========================================
  Files         200      201       +1     
  Lines       17049    17420     +371     
==========================================
+ Hits         9599     9671      +72     
- Misses       7450     7749     +299     
Impacted Files Coverage Δ
...c/syntax/parser/expression/primary/template/mod.rs 52.77% <0.00%> (-23.09%) ⬇️
...src/syntax/ast/node/iteration/continue_node/mod.rs 80.00% <0.00%> (-20.00%) ⬇️
boa/src/syntax/ast/node/identifier/mod.rs 57.14% <0.00%> (-17.86%) ⬇️
...c/syntax/ast/node/declaration/function_expr/mod.rs 57.14% <0.00%> (-17.86%) ⬇️
...src/syntax/ast/node/iteration/do_while_loop/mod.rs 54.54% <0.00%> (-16.89%) ⬇️
...oa/src/syntax/ast/node/iteration/while_loop/mod.rs 54.54% <0.00%> (-15.46%) ⬇️
...src/syntax/parser/expression/left_hand_side/mod.rs 76.92% <0.00%> (-13.99%) ⬇️
boa/src/syntax/parser/expression/await_expr.rs 87.50% <0.00%> (-12.50%) ⬇️
...a/src/syntax/parser/expression/assignment/yield.rs 61.53% <0.00%> (-12.15%) ⬇️
...a/src/syntax/ast/node/iteration/for_in_loop/mod.rs 53.84% <0.00%> (-11.38%) ⬇️
... and 117 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d831ff3...e69d80c. Read the comment docs.

@boa-dev boa-dev deleted a comment from github-actions bot Jan 27, 2022
@boa-dev boa-dev deleted a comment from github-actions bot Jan 27, 2022
@github-actions
Copy link

Benchmark for 7494616

Click to view benchmark
Test Base PR %
Arithmetic operations (Compiler) 385.1±0.85ns 383.0±0.81ns -0.55%
Arithmetic operations (Execution) 1762.8±4.01ns 1762.9±2.05ns +0.01%
Arithmetic operations (Parser) 6.1±0.06µs 6.1±0.04µs 0.00%
Array access (Compiler) 1006.6±3.14ns 1005.9±4.40ns -0.07%
Array access (Execution) 10.7±0.03µs 10.9±0.03µs +1.87%
Array access (Parser) 13.2±0.09µs 13.0±0.06µs -1.52%
Array creation (Compiler) 1399.4±3.28ns 1442.2±2.54ns +3.06%
Array creation (Execution) 3.2±0.04ms 3.3±0.00ms +3.13%
Array creation (Parser) 14.5±0.04µs 14.4±0.06µs -0.69%
Array pop (Compiler) 3.0±0.01µs 3.0±0.04µs 0.00%
Array pop (Execution) 1365.5±16.03µs 1333.1±8.36µs -2.37%
Array pop (Parser) 148.9±0.20µs 149.2±0.21µs +0.20%
Boolean Object Access (Compiler) 843.0±1.99ns 846.8±1.78ns +0.45%
Boolean Object Access (Execution) 6.8±0.03µs 6.9±0.10µs +1.47%
Boolean Object Access (Parser) 14.7±0.09µs 14.9±0.05µs +1.36%
Clean js (Compiler) 2.9±0.01µs 3.0±0.01µs +3.45%
Clean js (Execution) 1486.5±20.56µs 1503.6±11.12µs +1.15%
Clean js (Parser) 31.0±0.35µs 30.8±0.09µs -0.65%
Create Realm 324.2±2.91ns 325.2±2.12ns +0.31%
Dynamic Object Property Access (Compiler) 1384.3±2.63ns 1402.5±2.40ns +1.31%
Dynamic Object Property Access (Execution) 6.6±0.03µs 6.8±0.04µs +3.03%
Dynamic Object Property Access (Parser) 11.6±0.03µs 11.6±0.06µs 0.00%
Fibonacci (Compiler) 1841.3±4.75ns 1876.4±7.43ns +1.91%
Fibonacci (Execution) 2.8±0.00ms 2.9±0.01ms +3.57%
Fibonacci (Parser) 17.6±0.17µs 17.3±0.11µs -1.70%
For loop (Compiler) 1539.1±40.09ns 1550.1±4.89ns +0.71%
For loop (Execution) 43.1±0.22µs 43.2±0.21µs +0.23%
For loop (Parser) 15.1±0.20µs 14.8±0.08µs -1.99%
Mini js (Compiler) 2.9±0.01µs 2.9±0.01µs 0.00%
Mini js (Execution) 1387.8±27.99µs 1410.9±7.97µs +1.66%
Mini js (Parser) 27.3±0.34µs 27.1±0.10µs -0.73%
Number Object Access (Compiler) 771.7±2.94ns 777.5±1.88ns +0.75%
Number Object Access (Execution) 5.3±0.09µs 5.4±0.06µs +1.89%
Number Object Access (Parser) 11.9±0.25µs 11.6±0.08µs -2.52%
Object Creation (Compiler) 1180.5±3.34ns 1183.9±5.12ns +0.29%
Object Creation (Execution) 6.0±0.02µs 6.1±0.04µs +1.67%
Object Creation (Parser) 10.1±0.10µs 10.0±0.02µs -0.99%
RegExp (Compiler) 1315.9±3.75ns 1329.1±3.95ns +1.00%
RegExp (Execution) 12.9±0.04µs 13.0±0.03µs +0.78%
RegExp (Parser) 11.0±0.04µs 10.8±0.04µs -1.82%
RegExp Creation (Compiler) 1178.4±4.04ns 1196.1±5.57ns +1.50%
RegExp Creation (Execution) 9.6±0.04µs 9.7±0.05µs +1.04%
RegExp Creation (Parser) 9.1±0.05µs 9.1±0.04µs 0.00%
RegExp Literal (Compiler) 1323.7±3.01ns 1325.6±4.86ns +0.14%
RegExp Literal (Execution) 13.0±0.03µs 13.0±0.04µs 0.00%
RegExp Literal (Parser) 8.9±0.06µs 8.9±0.03µs 0.00%
RegExp Literal Creation (Compiler) 1178.2±3.81ns 1180.7±4.04ns +0.21%
RegExp Literal Creation (Execution) 9.7±0.05µs 9.7±0.05µs 0.00%
RegExp Literal Creation (Parser) 7.1±0.05µs 7.2±0.03µs +1.41%
Static Object Property Access (Compiler) 1209.1±5.38ns 1239.7±6.06ns +2.53%
Static Object Property Access (Execution) 6.2±0.03µs 6.4±0.04µs +3.23%
Static Object Property Access (Parser) 10.7±0.07µs 10.7±0.02µs 0.00%
String Object Access (Compiler) 1196.3±4.25ns 1199.4±12.85ns +0.26%
String Object Access (Execution) 8.5±0.02µs 8.8±0.09µs +3.53%
String Object Access (Parser) 14.3±0.03µs 14.4±0.06µs +0.70%
String comparison (Compiler) 1669.4±11.60ns 1692.0±10.89ns +1.35%
String comparison (Execution) 7.4±0.04µs 7.5±0.03µs +1.35%
String comparison (Parser) 11.8±0.11µs 11.7±0.02µs -0.85%
String concatenation (Compiler) 1290.0±5.33ns 1289.6±7.59ns -0.03%
String concatenation (Execution) 6.2±0.02µs 6.3±0.04µs +1.61%
String concatenation (Parser) 8.2±0.09µs 8.2±0.07µs 0.00%
String copy (Compiler) 1033.7±3.44ns 1030.5±3.23ns -0.31%
String copy (Execution) 5.3±0.02µs 5.3±0.02µs 0.00%
String copy (Parser) 6.2±0.09µs 6.1±0.02µs -1.61%
Symbols (Compiler) 708.1±1.35ns 703.2±1.50ns -0.69%
Symbols (Execution) 4.9±0.01µs 4.9±0.01µs 0.00%
Symbols (Parser) 4.6±0.01µs 4.7±0.03µs +2.17%

@Razican
Copy link
Member Author

Razican commented Jan 29, 2022

This is now ready to be merged :)

@Razican Razican added this to the v0.14.0 milestone Jan 29, 2022
@Razican Razican changed the title Trying the new action fixes for PR benchmarks Update the PR benchmarks action Jan 29, 2022
Copy link
Member

@jedel1043 jedel1043 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, I realized the criterion-compare-action repo didn't have a version bump to 3.0.1 on the package.json (https://github.com/boa-dev/criterion-compare-action/blob/b12b240867eafcd503e063c17db8db3c4ab50d7e/package.json#L3)

@@ -21,8 +21,8 @@ jobs:
~/.cargo/git
~/.cargo/registry
key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
- uses: boa-dev/criterion-compare-action@master
- uses: boa-dev/criterion-compare-action@v3.0.1
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason for pinning the action to the version 3.0.1? We could use main to avoid having to bump the version on every new change.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The idea behind this is to follow the nomenclature with the rest of actions. This action is now being used by more projects, not only Boa, so we should use the released versions. That way we can also develop the action in the main branch as needed, and release new versions with new features. Also, @dependabot will create PRs to update the action if we do it like this.

@Razican
Copy link
Member Author

Razican commented Jan 29, 2022

Also, I realized the criterion-compare-action repo didn't have a version bump to 3.0.1 on the package.json (https://github.com/boa-dev/criterion-compare-action/blob/b12b240867eafcd503e063c17db8db3c4ab50d7e/package.json#L3)

I will update this, I forgot it :/

Copy link
Member

@jedel1043 jedel1043 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All good then!

@Razican
Copy link
Member Author

Razican commented Jan 29, 2022

All good then!

Fixed the version, released a new one and updated the GitHub action.

@github-actions
Copy link

Benchmark for c9be5af

Click to view benchmark
Test Base PR %
Arithmetic operations (Compiler) 419.7±2.50ns 399.4±0.56ns -4.84%
Arithmetic operations (Execution) 1992.4±4.70ns 1989.3±2.06ns -0.16%
Arithmetic operations (Parser) 5.8±0.01µs 5.8±0.02µs 0.00%
Array access (Compiler) 1007.1±4.58ns 997.9±4.46ns -0.91%
Array access (Execution) 10.5±0.03µs 10.6±0.06µs +0.95%
Array access (Parser) 12.1±0.02µs 12.1±0.09µs 0.00%
Array creation (Compiler) 1364.1±3.59ns 1349.8±2.67ns -1.05%
Array creation (Execution) 3.2±0.01ms 3.2±0.01ms 0.00%
Array creation (Parser) 13.7±0.04µs 13.6±0.02µs -0.73%
Array pop (Compiler) 3.2±0.01µs 3.2±0.01µs 0.00%
Array pop (Execution) 1334.7±4.53µs 1322.0±4.32µs -0.95%
Array pop (Parser) 142.5±0.19µs 139.6±0.11µs -2.04%
Boolean Object Access (Compiler) 863.9±1.52ns 856.6±1.00ns -0.85%
Boolean Object Access (Execution) 6.8±0.01µs 6.8±0.02µs 0.00%
Boolean Object Access (Parser) 14.3±0.03µs 14.3±0.03µs 0.00%
Clean js (Compiler) 2.8±0.01µs 2.9±0.01µs +3.57%
Clean js (Execution) 1486.0±14.84µs 1473.7±21.62µs -0.83%
Clean js (Parser) 29.6±0.11µs 29.2±0.03µs -1.35%
Create Realm 337.8±0.38ns 337.0±0.31ns -0.24%
Dynamic Object Property Access (Compiler) 1333.6±4.31ns 1335.2±4.23ns +0.12%
Dynamic Object Property Access (Execution) 6.9±0.05µs 6.9±0.05µs 0.00%
Dynamic Object Property Access (Parser) 11.0±0.02µs 10.7±0.01µs -2.73%
Fibonacci (Compiler) 1799.8±5.77ns 1848.9±6.12ns +2.73%
Fibonacci (Execution) 2.8±0.01ms 2.8±0.01ms 0.00%
Fibonacci (Parser) 16.3±0.07µs 16.2±0.03µs -0.61%
For loop (Compiler) 1469.0±12.50ns 1466.7±9.37ns -0.16%
For loop (Execution) 45.2±0.11µs 45.0±0.13µs -0.44%
For loop (Parser) 14.3±0.07µs 14.0±0.02µs -2.10%
Mini js (Compiler) 2.9±0.01µs 2.9±0.01µs 0.00%
Mini js (Execution) 1380.2±14.69µs 1373.1±26.87µs -0.51%
Mini js (Parser) 26.0±0.13µs 25.7±0.02µs -1.15%
Number Object Access (Compiler) 770.8±1.94ns 771.1±1.64ns +0.04%
Number Object Access (Execution) 5.3±0.02µs 5.4±0.01µs +1.89%
Number Object Access (Parser) 11.5±0.03µs 11.4±0.03µs -0.87%
Object Creation (Compiler) 1151.2±4.60ns 1145.9±2.35ns -0.46%
Object Creation (Execution) 6.2±0.03µs 6.3±0.02µs +1.61%
Object Creation (Parser) 9.4±0.02µs 9.3±0.05µs -1.06%
RegExp (Compiler) 1278.5±4.18ns 1305.0±3.96ns +2.07%
RegExp (Execution) 12.6±0.04µs 12.7±0.03µs +0.79%
RegExp (Parser) 10.4±0.04µs 10.2±0.01µs -1.92%
RegExp Creation (Compiler) 1168.8±5.49ns 1145.5±2.29ns -1.99%
RegExp Creation (Execution) 9.4±0.05µs 9.6±0.07µs +2.13%
RegExp Creation (Parser) 8.7±0.02µs 8.6±0.01µs -1.15%
RegExp Literal (Compiler) 1280.6±4.08ns 1275.4±3.59ns -0.41%
RegExp Literal (Execution) 12.6±0.04µs 12.7±0.02µs +0.79%
RegExp Literal (Parser) 8.3±0.01µs 8.2±0.01µs -1.20%
RegExp Literal Creation (Compiler) 1133.6±1.52ns 1144.3±3.29ns +0.94%
RegExp Literal Creation (Execution) 9.5±0.11µs 9.5±0.05µs 0.00%
RegExp Literal Creation (Parser) 6.4±0.01µs 6.5±0.01µs +1.56%
Static Object Property Access (Compiler) 1173.6±7.34ns 1171.7±3.95ns -0.16%
Static Object Property Access (Execution) 6.4±0.03µs 6.5±0.04µs +1.56%
Static Object Property Access (Parser) 10.1±0.02µs 10.0±0.01µs -0.99%
String Object Access (Compiler) 1201.8±9.78ns 1208.0±11.63ns +0.52%
String Object Access (Execution) 8.6±0.02µs 8.6±0.01µs 0.00%
String Object Access (Parser) 14.3±0.03µs 14.1±0.04µs -1.40%
String comparison (Compiler) 1656.4±9.88ns 1657.9±15.30ns +0.09%
String comparison (Execution) 7.4±0.03µs 7.4±0.02µs 0.00%
String comparison (Parser) 11.3±0.02µs 11.1±0.05µs -1.77%
String concatenation (Compiler) 1287.6±4.50ns 1280.0±4.19ns -0.59%
String concatenation (Execution) 6.2±0.04µs 6.3±0.04µs +1.61%
String concatenation (Parser) 7.8±0.02µs 7.6±0.02µs -2.56%
String copy (Compiler) 1016.8±4.06ns 1018.1±4.74ns +0.13%
String copy (Execution) 5.3±0.01µs 5.4±0.02µs +1.89%
String copy (Parser) 5.8±0.01µs 5.7±0.02µs -1.72%
Symbols (Compiler) 696.6±1.18ns 680.6±1.48ns -2.30%
Symbols (Execution) 5.0±0.01µs 5.0±0.01µs 0.00%
Symbols (Parser) 4.3±0.02µs 4.4±0.01µs +2.33%

@github-actions
Copy link

Benchmark for 30df9fb

Click to view benchmark
Test Base PR %
Arithmetic operations (Compiler) 461.5±11.28ns 487.2±3.32ns +5.57%
Arithmetic operations (Execution) 2.4±0.02µs 2.3±0.03µs -4.17%
Arithmetic operations (Parser) 6.2±0.12µs 6.5±0.05µs +4.84%
Array access (Compiler) 1163.0±18.93ns 1150.3±23.69ns -1.09%
Array access (Execution) 12.2±0.32µs 12.5±0.28µs +2.46%
Array access (Parser) 13.4±0.14µs 13.4±0.06µs 0.00%
Array creation (Compiler) 1575.0±26.23ns 1575.2±30.46ns +0.01%
Array creation (Execution) 3.8±0.07ms 3.9±0.03ms +2.63%
Array creation (Parser) 14.9±0.13µs 15.1±0.26µs +1.34%
Array pop (Compiler) 3.9±0.04µs 3.8±0.14µs -2.56%
Array pop (Execution) 1561.4±22.72µs 1580.0±12.83µs +1.19%
Array pop (Parser) 164.1±1.06µs 165.8±1.63µs +1.04%
Boolean Object Access (Compiler) 961.2±21.59ns 975.0±14.25ns +1.44%
Boolean Object Access (Execution) 8.0±0.09µs 7.9±0.11µs -1.25%
Boolean Object Access (Parser) 15.7±0.29µs 16.0±0.13µs +1.91%
Clean js (Compiler) 3.3±0.06µs 3.4±0.04µs +3.03%
Clean js (Execution) 1770.4±16.01µs 1774.4±34.67µs +0.23%
Clean js (Parser) 31.7±0.64µs 32.5±0.27µs +2.52%
Create Realm 401.9±6.58ns 402.8±4.45ns +0.22%
Dynamic Object Property Access (Compiler) 1565.9±27.53ns 1684.5±109.28ns +7.57%
Dynamic Object Property Access (Execution) 8.2±0.10µs 8.1±0.19µs -1.22%
Dynamic Object Property Access (Parser) 11.7±0.10µs 11.6±0.24µs -0.85%
Fibonacci (Compiler) 2.1±0.04µs 2.2±0.02µs +4.76%
Fibonacci (Execution) 3.3±0.03ms 3.3±0.04ms 0.00%
Fibonacci (Parser) 17.9±0.18µs 17.5±0.32µs -2.23%
For loop (Compiler) 1748.6±15.17ns 1866.6±129.86ns +6.75%
For loop (Execution) 53.8±0.53µs 53.8±0.25µs 0.00%
For loop (Parser) 15.5±0.16µs 15.8±0.22µs +1.94%
Mini js (Compiler) 3.3±0.05µs 3.4±0.04µs +3.03%
Mini js (Execution) 1646.2±11.54µs 1711.5±90.60µs +3.97%
Mini js (Parser) 28.4±0.31µs 28.5±0.23µs +0.35%
Number Object Access (Compiler) 886.5±13.71ns 878.7±14.77ns -0.88%
Number Object Access (Execution) 6.3±0.09µs 6.4±0.10µs +1.59%
Number Object Access (Parser) 12.2±0.24µs 12.5±0.11µs +2.46%
Object Creation (Compiler) 1345.4±25.16ns 1376.2±9.54ns +2.29%
Object Creation (Execution) 7.4±0.12µs 7.3±0.09µs -1.35%
Object Creation (Parser) 10.2±0.12µs 10.2±0.20µs 0.00%
RegExp (Compiler) 1494.4±26.15ns 1525.8±31.23ns +2.10%
RegExp (Execution) 14.6±0.20µs 14.7±0.27µs +0.68%
RegExp (Parser) 11.1±0.10µs 11.4±0.07µs +2.70%
RegExp Creation (Compiler) 1356.3±18.91ns 1467.4±101.18ns +8.19%
RegExp Creation (Execution) 11.0±0.17µs 11.1±0.19µs +0.91%
RegExp Creation (Parser) 9.3±0.10µs 9.4±0.17µs +1.08%
RegExp Literal (Compiler) 1504.0±29.82ns 1514.1±17.03ns +0.67%
RegExp Literal (Execution) 14.3±0.27µs 14.7±0.30µs +2.80%
RegExp Literal (Parser) 8.9±0.10µs 9.1±0.11µs +2.25%
RegExp Literal Creation (Compiler) 1323.8±22.79ns 1368.0±10.50ns +3.34%
RegExp Literal Creation (Execution) 11.0±0.26µs 11.1±0.18µs +0.91%
RegExp Literal Creation (Parser) 7.1±0.05µs 7.0±0.09µs -1.41%
Static Object Property Access (Compiler) 1373.3±18.23ns 1379.8±25.15ns +0.47%
Static Object Property Access (Execution) 7.6±0.17µs 7.5±0.15µs -1.32%
Static Object Property Access (Parser) 10.9±0.09µs 10.8±0.16µs -0.92%
String Object Access (Compiler) 1402.7±25.16ns 1418.0±15.36ns +1.09%
String Object Access (Execution) 10.2±0.17µs 10.2±0.17µs 0.00%
String Object Access (Parser) 15.5±0.20µs 15.9±0.15µs +2.58%
String comparison (Compiler) 1938.2±47.50ns 1936.8±35.76ns -0.07%
String comparison (Execution) 8.8±0.15µs 8.7±0.11µs -1.14%
String comparison (Parser) 12.2±0.17µs 12.6±0.09µs +3.28%
String concatenation (Compiler) 1514.4±23.18ns 1533.1±20.79ns +1.23%
String concatenation (Execution) 7.4±0.12µs 7.4±0.17µs 0.00%
String concatenation (Parser) 8.3±0.12µs 8.6±0.05µs +3.61%
String copy (Compiler) 1185.2±20.58ns 1184.9±20.50ns -0.03%
String copy (Execution) 6.3±0.10µs 6.3±0.11µs 0.00%
String copy (Parser) 6.2±0.07µs 6.7±0.51µs +8.06%
Symbols (Compiler) 818.6±6.32ns 810.1±7.21ns -1.04%
Symbols (Execution) 5.8±0.12µs 5.9±0.07µs +1.72%
Symbols (Parser) 4.7±0.03µs 4.6±0.03µs -2.13%

@Razican
Copy link
Member Author

Razican commented Jan 29, 2022

bors r+

bors bot pushed a commit that referenced this pull request Jan 29, 2022
This is just for debugging for now. If we find out a fix, we can merge it.
@bors
Copy link

bors bot commented Jan 29, 2022

Pull request successfully merged into main.

Build succeeded:

@bors bors bot changed the title Update the PR benchmarks action [Merged by Bors] - Update the PR benchmarks action Jan 29, 2022
@bors bors bot closed this Jan 29, 2022
@bors bors bot deleted the fix_pr_benchmarks branch January 29, 2022 17:45
@RageKnify RageKnify added the Internal Category for changelog label Mar 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
benchmark Issues and PRs related to the benchmark subsystem. Internal Category for changelog
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants