Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ARROW-12330: [Developer] Restore values at counters column of Archery benchmark #9985

Closed
wants to merge 4 commits into from

Conversation

kiszk
Copy link
Member

@kiszk kiszk commented Apr 11, 2021

This PR restores values at counters column of Archery benchmark. #9140 always suppressed values at counters column regardless --no-counter. In addition, this PR stores counters value into json file.

Before

% archery benchmark diff --benchmark-filter="SetBitsTo" --output=head2.json HEAD HEAD~1
...
---------------------------------------------------------------------------
Benchmark                 Time             CPU   Iterations UserCounters...
---------------------------------------------------------------------------
SetBitsTo/2            8.15 ns         8.15 ns     81991087 bytes_per_second=234.044M/s
SetBitsTo/16           7.78 ns         7.78 ns     89928878 bytes_per_second=1.91429G/s
SetBitsTo/1024         13.9 ns         13.9 ns     50372172 bytes_per_second=68.6182G/s
SetBitsTo/131072       3508 ns         3508 ns       199335 bytes_per_second=34.7944G/s
----------------------------------------------------------------------
Non-regressions: (4)
----------------------------------------------------------------------
        benchmark         baseline        contender  change % counters
     SetBitsTo/16    1.877 GiB/sec    1.914 GiB/sec     1.975       {}
      SetBitsTo/2  230.566 MiB/sec  234.044 MiB/sec     1.509       {}
 SetBitsTo/131072   34.722 GiB/sec   34.794 GiB/sec     0.207       {}
   SetBitsTo/1024   68.593 GiB/sec   68.618 GiB/sec     0.037       {}

After

---------------------------------------------------------------------------
Benchmark                 Time             CPU   Iterations UserCounters...
---------------------------------------------------------------------------
SetBitsTo/2            8.39 ns         8.39 ns     81980047 bytes_per_second=227.438M/s
SetBitsTo/16           7.88 ns         7.88 ns     84936624 bytes_per_second=1.89105G/s
SetBitsTo/1024         13.9 ns         13.9 ns     50376587 bytes_per_second=68.6064G/s
SetBitsTo/131072       3513 ns         3513 ns       200598 bytes_per_second=34.7447G/s
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Non-regressions: (4)
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
        benchmark         baseline        contender  change %                                                                                                       counters
      SetBitsTo/2  227.438 MiB/sec  235.984 MiB/sec     3.757     {'run_name': 'SetBitsTo/2', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 81980047}
     SetBitsTo/16    1.891 GiB/sec    1.913 GiB/sec     1.137    {'run_name': 'SetBitsTo/16', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 84936624}
 SetBitsTo/131072   34.745 GiB/sec   34.771 GiB/sec     0.075  {'run_name': 'SetBitsTo/131072', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 200598}
   SetBitsTo/1024   68.606 GiB/sec   68.624 GiB/sec     0.026  {'run_name': 'SetBitsTo/1024', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 50376587}

@github-actions
Copy link

@kszucs
Copy link
Member

kszucs commented Apr 13, 2021

@kiszk thanks for working on this! Could you please fix the archery unittests?

@kiszk
Copy link
Member Author

kiszk commented Apr 13, 2021

Of course, I will fix it in a few days.

@kszucs kszucs closed this in fb892c6 Apr 14, 2021
Dandandan pushed a commit to Dandandan/arrow that referenced this pull request Apr 14, 2021
… benchmark

This PR restores values at `counters` column of Archery benchmark. apache#9140 always suppressed values at `counters` column regardless `--no-counter`. In addition, this PR stores `counters` value into json file.

Before
```
% archery benchmark diff --benchmark-filter="SetBitsTo" --output=head2.json HEAD HEAD~1
...
---------------------------------------------------------------------------
Benchmark                 Time             CPU   Iterations UserCounters...
---------------------------------------------------------------------------
SetBitsTo/2            8.15 ns         8.15 ns     81991087 bytes_per_second=234.044M/s
SetBitsTo/16           7.78 ns         7.78 ns     89928878 bytes_per_second=1.91429G/s
SetBitsTo/1024         13.9 ns         13.9 ns     50372172 bytes_per_second=68.6182G/s
SetBitsTo/131072       3508 ns         3508 ns       199335 bytes_per_second=34.7944G/s
----------------------------------------------------------------------
Non-regressions: (4)
----------------------------------------------------------------------
        benchmark         baseline        contender  change % counters
     SetBitsTo/16    1.877 GiB/sec    1.914 GiB/sec     1.975       {}
      SetBitsTo/2  230.566 MiB/sec  234.044 MiB/sec     1.509       {}
 SetBitsTo/131072   34.722 GiB/sec   34.794 GiB/sec     0.207       {}
   SetBitsTo/1024   68.593 GiB/sec   68.618 GiB/sec     0.037       {}
```

After
```
---------------------------------------------------------------------------
Benchmark                 Time             CPU   Iterations UserCounters...
---------------------------------------------------------------------------
SetBitsTo/2            8.39 ns         8.39 ns     81980047 bytes_per_second=227.438M/s
SetBitsTo/16           7.88 ns         7.88 ns     84936624 bytes_per_second=1.89105G/s
SetBitsTo/1024         13.9 ns         13.9 ns     50376587 bytes_per_second=68.6064G/s
SetBitsTo/131072       3513 ns         3513 ns       200598 bytes_per_second=34.7447G/s
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Non-regressions: (4)
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
        benchmark         baseline        contender  change %                                                                                                       counters
      SetBitsTo/2  227.438 MiB/sec  235.984 MiB/sec     3.757     {'run_name': 'SetBitsTo/2', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 81980047}
     SetBitsTo/16    1.891 GiB/sec    1.913 GiB/sec     1.137    {'run_name': 'SetBitsTo/16', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 84936624}
 SetBitsTo/131072   34.745 GiB/sec   34.771 GiB/sec     0.075  {'run_name': 'SetBitsTo/131072', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 200598}
   SetBitsTo/1024   68.606 GiB/sec   68.624 GiB/sec     0.026  {'run_name': 'SetBitsTo/1024', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 50376587}
```

Closes apache#9985 from kiszk/ARROW-12330

Authored-by: Kazuaki Ishizaki <ishizaki@jp.ibm.com>
Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
GeorgeAp pushed a commit to sirensolutions/arrow that referenced this pull request Jun 7, 2021
… benchmark

This PR restores values at `counters` column of Archery benchmark. apache#9140 always suppressed values at `counters` column regardless `--no-counter`. In addition, this PR stores `counters` value into json file.

Before
```
% archery benchmark diff --benchmark-filter="SetBitsTo" --output=head2.json HEAD HEAD~1
...
---------------------------------------------------------------------------
Benchmark                 Time             CPU   Iterations UserCounters...
---------------------------------------------------------------------------
SetBitsTo/2            8.15 ns         8.15 ns     81991087 bytes_per_second=234.044M/s
SetBitsTo/16           7.78 ns         7.78 ns     89928878 bytes_per_second=1.91429G/s
SetBitsTo/1024         13.9 ns         13.9 ns     50372172 bytes_per_second=68.6182G/s
SetBitsTo/131072       3508 ns         3508 ns       199335 bytes_per_second=34.7944G/s
----------------------------------------------------------------------
Non-regressions: (4)
----------------------------------------------------------------------
        benchmark         baseline        contender  change % counters
     SetBitsTo/16    1.877 GiB/sec    1.914 GiB/sec     1.975       {}
      SetBitsTo/2  230.566 MiB/sec  234.044 MiB/sec     1.509       {}
 SetBitsTo/131072   34.722 GiB/sec   34.794 GiB/sec     0.207       {}
   SetBitsTo/1024   68.593 GiB/sec   68.618 GiB/sec     0.037       {}
```

After
```
---------------------------------------------------------------------------
Benchmark                 Time             CPU   Iterations UserCounters...
---------------------------------------------------------------------------
SetBitsTo/2            8.39 ns         8.39 ns     81980047 bytes_per_second=227.438M/s
SetBitsTo/16           7.88 ns         7.88 ns     84936624 bytes_per_second=1.89105G/s
SetBitsTo/1024         13.9 ns         13.9 ns     50376587 bytes_per_second=68.6064G/s
SetBitsTo/131072       3513 ns         3513 ns       200598 bytes_per_second=34.7447G/s
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Non-regressions: (4)
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
        benchmark         baseline        contender  change %                                                                                                       counters
      SetBitsTo/2  227.438 MiB/sec  235.984 MiB/sec     3.757     {'run_name': 'SetBitsTo/2', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 81980047}
     SetBitsTo/16    1.891 GiB/sec    1.913 GiB/sec     1.137    {'run_name': 'SetBitsTo/16', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 84936624}
 SetBitsTo/131072   34.745 GiB/sec   34.771 GiB/sec     0.075  {'run_name': 'SetBitsTo/131072', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 200598}
   SetBitsTo/1024   68.606 GiB/sec   68.624 GiB/sec     0.026  {'run_name': 'SetBitsTo/1024', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 50376587}
```

Closes apache#9985 from kiszk/ARROW-12330

Authored-by: Kazuaki Ishizaki <ishizaki@jp.ibm.com>
Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
michalursa pushed a commit to michalursa/arrow that referenced this pull request Jun 10, 2021
… benchmark

This PR restores values at `counters` column of Archery benchmark. apache#9140 always suppressed values at `counters` column regardless `--no-counter`. In addition, this PR stores `counters` value into json file.

Before
```
% archery benchmark diff --benchmark-filter="SetBitsTo" --output=head2.json HEAD HEAD~1
...
---------------------------------------------------------------------------
Benchmark                 Time             CPU   Iterations UserCounters...
---------------------------------------------------------------------------
SetBitsTo/2            8.15 ns         8.15 ns     81991087 bytes_per_second=234.044M/s
SetBitsTo/16           7.78 ns         7.78 ns     89928878 bytes_per_second=1.91429G/s
SetBitsTo/1024         13.9 ns         13.9 ns     50372172 bytes_per_second=68.6182G/s
SetBitsTo/131072       3508 ns         3508 ns       199335 bytes_per_second=34.7944G/s
----------------------------------------------------------------------
Non-regressions: (4)
----------------------------------------------------------------------
        benchmark         baseline        contender  change % counters
     SetBitsTo/16    1.877 GiB/sec    1.914 GiB/sec     1.975       {}
      SetBitsTo/2  230.566 MiB/sec  234.044 MiB/sec     1.509       {}
 SetBitsTo/131072   34.722 GiB/sec   34.794 GiB/sec     0.207       {}
   SetBitsTo/1024   68.593 GiB/sec   68.618 GiB/sec     0.037       {}
```

After
```
---------------------------------------------------------------------------
Benchmark                 Time             CPU   Iterations UserCounters...
---------------------------------------------------------------------------
SetBitsTo/2            8.39 ns         8.39 ns     81980047 bytes_per_second=227.438M/s
SetBitsTo/16           7.88 ns         7.88 ns     84936624 bytes_per_second=1.89105G/s
SetBitsTo/1024         13.9 ns         13.9 ns     50376587 bytes_per_second=68.6064G/s
SetBitsTo/131072       3513 ns         3513 ns       200598 bytes_per_second=34.7447G/s
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Non-regressions: (4)
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
        benchmark         baseline        contender  change %                                                                                                       counters
      SetBitsTo/2  227.438 MiB/sec  235.984 MiB/sec     3.757     {'run_name': 'SetBitsTo/2', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 81980047}
     SetBitsTo/16    1.891 GiB/sec    1.913 GiB/sec     1.137    {'run_name': 'SetBitsTo/16', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 84936624}
 SetBitsTo/131072   34.745 GiB/sec   34.771 GiB/sec     0.075  {'run_name': 'SetBitsTo/131072', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 200598}
   SetBitsTo/1024   68.606 GiB/sec   68.624 GiB/sec     0.026  {'run_name': 'SetBitsTo/1024', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 50376587}
```

Closes apache#9985 from kiszk/ARROW-12330

Authored-by: Kazuaki Ishizaki <ishizaki@jp.ibm.com>
Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
michalursa pushed a commit to michalursa/arrow that referenced this pull request Jun 13, 2021
… benchmark

This PR restores values at `counters` column of Archery benchmark. apache#9140 always suppressed values at `counters` column regardless `--no-counter`. In addition, this PR stores `counters` value into json file.

Before
```
% archery benchmark diff --benchmark-filter="SetBitsTo" --output=head2.json HEAD HEAD~1
...
---------------------------------------------------------------------------
Benchmark                 Time             CPU   Iterations UserCounters...
---------------------------------------------------------------------------
SetBitsTo/2            8.15 ns         8.15 ns     81991087 bytes_per_second=234.044M/s
SetBitsTo/16           7.78 ns         7.78 ns     89928878 bytes_per_second=1.91429G/s
SetBitsTo/1024         13.9 ns         13.9 ns     50372172 bytes_per_second=68.6182G/s
SetBitsTo/131072       3508 ns         3508 ns       199335 bytes_per_second=34.7944G/s
----------------------------------------------------------------------
Non-regressions: (4)
----------------------------------------------------------------------
        benchmark         baseline        contender  change % counters
     SetBitsTo/16    1.877 GiB/sec    1.914 GiB/sec     1.975       {}
      SetBitsTo/2  230.566 MiB/sec  234.044 MiB/sec     1.509       {}
 SetBitsTo/131072   34.722 GiB/sec   34.794 GiB/sec     0.207       {}
   SetBitsTo/1024   68.593 GiB/sec   68.618 GiB/sec     0.037       {}
```

After
```
---------------------------------------------------------------------------
Benchmark                 Time             CPU   Iterations UserCounters...
---------------------------------------------------------------------------
SetBitsTo/2            8.39 ns         8.39 ns     81980047 bytes_per_second=227.438M/s
SetBitsTo/16           7.88 ns         7.88 ns     84936624 bytes_per_second=1.89105G/s
SetBitsTo/1024         13.9 ns         13.9 ns     50376587 bytes_per_second=68.6064G/s
SetBitsTo/131072       3513 ns         3513 ns       200598 bytes_per_second=34.7447G/s
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Non-regressions: (4)
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
        benchmark         baseline        contender  change %                                                                                                       counters
      SetBitsTo/2  227.438 MiB/sec  235.984 MiB/sec     3.757     {'run_name': 'SetBitsTo/2', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 81980047}
     SetBitsTo/16    1.891 GiB/sec    1.913 GiB/sec     1.137    {'run_name': 'SetBitsTo/16', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 84936624}
 SetBitsTo/131072   34.745 GiB/sec   34.771 GiB/sec     0.075  {'run_name': 'SetBitsTo/131072', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 200598}
   SetBitsTo/1024   68.606 GiB/sec   68.624 GiB/sec     0.026  {'run_name': 'SetBitsTo/1024', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 50376587}
```

Closes apache#9985 from kiszk/ARROW-12330

Authored-by: Kazuaki Ishizaki <ishizaki@jp.ibm.com>
Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants