Added Leaderboard Entry for PPO

[ymahlau]

This adds a leaderboard entry for guided search using the action probabilities of a Model trained by using Proximal Policy Optimization (PPO).

Our full implementation can be found here: https://github.com/xtremey/ppo_compiler_gym

This method achieved a geometric mean of 1.070, which scores first place on the current leaderboard. Additionally, the walltime of the search is much lower than of other search algorithms on the leaderboard.

What is in this pull request?

• A write up of the method and csv-file containing the results.
• New Leaderboard Entry with the scores of our method

[codecov-commenter]

Codecov Report

Merging #580 (2c2ceae) into development (1d095e2) will decrease coverage by 0.26%.
The diff coverage is n/a.

@@               Coverage Diff               @@
##           development     #580      +/-   ##
===============================================
- Coverage        88.09%   87.82%   -0.27%     
===============================================
  Files              114      114              
  Lines             6693     6693              
===============================================
- Hits              5896     5878      -18     
- Misses             797      815      +18     

Impacted Files	Coverage Δ
...er_gym/third_party/inst2vec/inst2vec_preprocess.py	69.73% <0.00%> (-4.22%)	⬇️
...loop_tool/service/loop_tool_compilation_session.py	88.43% <0.00%> (-2.73%)	⬇️
compiler_gym/util/commands.py	83.33% <0.00%> (-2.39%)	⬇️
compiler_gym/envs/llvm/datasets/cbench.py	79.92% <0.00%> (-0.75%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1d095e2...2c2ceae. Read the comment docs.

[ChrisCummins]

Woah, cool! I'd love to see a PPO entry added to the leaderboard, and I like your neat trick about running a guided random search after.

However, I've not been able to validate your results. The command lines in your results.csv file are malformed (you need to add input.bc -o output.bc to the end of each), and, after fixing that, I was then unable to replicate the rewards achieved on my local linux box running CompilerGym 0.2.1.

$ python -m compiler_gym.bin.validate --env=llvm-ic-v0 results.csv 2>/dev/null
❌  cbench-v1/adpcm  Expected reward 1.0083798882681567 but received reward 0.994413407821229
❌  cbench-v1/stringsearch2  Expected reward 0.9962686567164181 but received reward 0.9440298507462687
❌  cbench-v1/rijndael  Expected reward 1.1092372556535077 but received reward 1.0785741663472594
❌  cbench-v1/ispell  Expected reward 1.028289936664321 but received reward 1.0216748768472907
❌  cbench-v1/qsort  Expected reward 1.136222910216718 but received reward 1.0773993808049536
...

Did you use the compiler_gym.leader.llvm_instcount.eval_llvm_instcount_policy helper to generate your results?

Cheers,
Chris

[ChrisCummins]

Nitpick: please strip the trailing whitespace in this file

[ChrisCummins]

It's entirely up to you, but if you'd prefer to list your three names here rather than the institution, that's totally fine :)

[ChrisCummins]

For us to accept this as open source your code must have a license (further reading here). Please add a LICENSE file to the linked repository, and state the name of the license here, e.g. "Is the approach Open Source? Yes, under Apache v2. Source code is here: ...".

Also, please link to an exact commit, such as: https://github.com/xtremey/ppo_compiler_gym/tree/99a677eaefdb33eddd01299aa683f8d0c14d8643
this enables you to make changes to your code without preventing people from reproducing your results here.

[ChrisCummins]

If I understand correctly, the PPO agent was trained on the same cbench programs that are used for the test set. Is that right? If so, this is an important distinction to make to make because it means that the agent is not being evaluated on its ability to generalize to an unseen program. Could you please add a few more details about the training process?

Since you are releasing your code, consider adding the hardware you used and commands needed to replicate the work. Take a look at this for an example: https://github.com/facebookresearch/CompilerGym/blob/development/leaderboard/llvm_instcount/dqn/README.md#experimental-setup

[ymahlau]

Thanks for the detailed feedback! The mistake was totally on our side. We used our own code to generate the results.csv file, but had a bug in the process.

We updated the Readme file and our own repository according to your feedback. If there is anything still missing or if anything could be improved please let us know.

Yannik

[ChrisCummins]

Hi Yannik,

Thanks for the quick fixes, and impressive results! I have reproduced your results on my end. Congrats on the top spot ⭐

$ python -m compiler_gym.bin.validate --env=llvm-ic-v0 results.csv                        
✅  cbench-v1/adpcm                           1.0084
✅  cbench-v1/rijndael                        1.1092
✅  cbench-v1/stringsearch2                   0.9963
✅  cbench-v1/ispell                          1.0283
✅  cbench-v1/lame                            1.0789
✅  cbench-v1/crc32                           1.0000
✅  cbench-v1/dijkstra                        0.9949
✅  cbench-v1/sha                             1.5251
✅  cbench-v1/qsort                           1.1362
✅  cbench-v1/bitcount                        1.0199
✅  cbench-v1/patricia                        1.0111
✅  cbench-v1/ghostscript                     1.0296
✅  cbench-v1/stringsearch                    1.0164
✅  cbench-v1/blowfish                        1.1018
✅  cbench-v1/gsm                             1.1296
✅  cbench-v1/susan                           1.0269
✅  cbench-v1/bzip2                           1.2212
✅  cbench-v1/tiffdither                      1.0489
✅  cbench-v1/tiffmedian                      1.0459
✅  cbench-v1/tiff2rgba                       1.0461
✅  cbench-v1/tiff2bw                         1.0446
✅  cbench-v1/jpeg-d                          1.0493
✅  cbench-v1/jpeg-c                          1.0468
----------------------------------------------------
Number of validated results: 23 of 23
Mean walltime per benchmark: 69.821s (std: 11.741s)
Geometric mean IrInstructionCountOz: 1.070 (std: 0.110)
python -m compiler_gym.bin.validate --env=llvm-ic-v0 results.csv  2121.22s user 124.60s system 1446% cpu 2:35.21 total

Cheers,
Chris