[llvm] Add API to construct a benchmark from clang invocation.

[ChrisCummins]

This adds a new API for constructing a benchmark from a command line invocation of a compiler. For example, given the command line:

$ gcc a.c src/b.c -DNDEBUG -lm -Oz -o foo.exe

The make_benchmark_from_command_line() method interprets this command line and rewrites it to one that emits an LLVM-IR file:

/path/to/compilergym/clang a.c src/b.c -DNDEG -lm -Oz -emit-llvm -c -o /path/to/benchmark.bc -Xclang disable-llvm-optmzns

It then loads the resulting bitcode for use as a benchmark. When the user is done playing around with the benchmark, they call a special compiler() method on the benchmark which re-invokes the original command line, but this time using the bitcode as input:

/path/to/compilergym/clang /path/to/benchmark.nc -DNDEG -lm -Oz -o foo.exe

Putting this all together, here is an end-to-end example using CompilerGym as a substitute for LLVM's pipeline:

>>> import compiler_gym
>>> env = compiler_gym.make("llvm-v0")
>>> benchmark = env.make_benchmark_from_command_line(                                                                                                                                                                                                                       
...    "gcc a.c src/b.c -DNDEBUG -lm -Oz -o foo.exe"
... )                                                                                                                                                                                                                                                                       
>>> for _ in range(100):
...     _, _, done, _ env.step(env.action_space.sample()
...     if done:
...         env.reset()
>>> env.benchmark.compile()
# Now the file foo.exe is compiled

The idea is to make it easier to integrate CompilerGym into an existing build system, as it is normally possible to dump a verbose log of all of the compiler invocations run during a build.

[codecov-commenter]

Codecov Report

Merging #577 (72ef9b1) into development (63dbfac) will increase coverage by 0.17%.
The diff coverage is 95.97%.

@@               Coverage Diff               @@
##           development     #577      +/-   ##
===============================================
+ Coverage        88.36%   88.54%   +0.17%     
===============================================
  Files              125      127       +2     
  Lines             7565     7761     +196     
===============================================
+ Hits              6685     6872     +187     
- Misses             880      889       +9     

Impacted Files	Coverage Δ
...piler_gym/envs/llvm/benchmark_from_command_line.py	92.68% <92.68%> (ø)
...ler_gym/third_party/gccinvocation/gccinvocation.py	95.91% <95.91%> (ø)
compiler_gym/envs/llvm/llvm_env.py	91.61% <98.27%> (+3.32%)	⬆️
compiler_gym/bin/validate.py	87.09% <100.00%> (ø)
compiler_gym/envs/llvm/__init__.py	100.00% <100.00%> (ø)
compiler_gym/service/connection.py	77.92% <0.00%> (-0.98%)	⬇️
...ompiler_gym/service/client_service_compiler_env.py	91.26% <0.00%> (+0.41%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 63dbfac...72ef9b1. Read the comment docs.

[mostafaelhoushi]

Super Cool!
I am wondering is that environment compatible with LlvmEnv or GccEnv? I mean can we directly call the existing observation or action functions on that environment?

[mostafaelhoushi]

The dataset or benchmark needs to be named clang-v0?

[mostafaelhoushi]

I am wondering if "gcc" works here... as down below we add -Xclang to the arguments

[ChrisCummins]

Yes, this function takes an argument replace_driver (default=True) that swaps out whatever the first argument is for clang. So this will work:

>>> env.make_benchmark_from_command_line(["gcc", "in.c"])

and will get translated to something like:

/path/to/compiler_gym/llvm/bin/clang in.c -c -emit-llvm -o - -Xclang disable-llvm-optmzns

but this will not:

>>> env.make_benchmark_from_command_line(["gcc", "in.c"], replace_driver=False)

The idea with replace_driver=False is that it allows you to use your own compiler to produce the input bitcode, providing your compiler produces compatible bitcode.

The downside is that it won't work with command line options that aren't the same between whatever compiler the user uses and our version of clang.

[ChrisCummins]

I'll extend the docstring to explain this better

[mostafaelhoushi]

so is this library literary gccinvokcation or is it also a generic C compiler invocator?

[ChrisCummins]

I've hacked on it to add support for clang by adding support for a few clang-specific flags, though I'm sure I've missed some

[ChrisCummins]

I don't want to change the name of the third-party library, though if my edits work well then I could submit them upstream and consider renaming it to "ccinvocation" or something

[hughleat]

LGTM.
Quick question though. How does it deal with mixed .c and .o files? E.g.:

gcc foo.c bar.o -o foo

My guess is that bar.o will be considered a source and that clang will barf trying to bitcode-ise it. Should those be filtered out for linking?

[ChrisCummins]

LGTM. Quick question though. How does it deal with mixed .c and .o files?

Good question! I added support so that .o files are excluded from the bitcode environment, but are kept for linking later.

The current implementation for this will not work if the CompilerGym backend and frontend are on different machines. To fix that, we need #325.

Cheers,
Chris