Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a wrapper to synchronously log environment state transitions to SQLite database #679

Merged

Conversation

ChrisCummins
Copy link
Contributor

This adds a compiler_gym.envs.wrappers.SynchronousSqliteLogger wrapper for an LLVM environment that logs all transitions to an sqlite database.

Wrap an existing LLVM environment and then use it as per normal:

>>> env = SynchronousSqliteLogger(
...     env=gym.make("llvm-autophase-ic-v0"),
...     db_path="example.db",
... )

Connect to the database file you specified:

$ sqlite3 example.db

There are two tables:

  1. States: records every unique combination of benchmark + actions. For each entry, records an identifying state ID, the episode reward, and whether the episode is terminated:
sqlite> .mode markdown
sqlite> .headers on
sqlite> select * from States limit 5;
|      benchmark_uri       | done | ir_instruction_count_oz_reward |                                 state_id | actions |
|--------------------------|------|--------------------------------|------------------------------------------|----------------|
| generator://csmith-v0/99 | 0    | 0.0                            | d625b874e58f6d357b816e21871297ac5c001cf0 |                |
| generator://csmith-v0/99 | 0    | 0.0                            | d625b874e58f6d357b816e21871297ac5c001cf0 | 31             |
| generator://csmith-v0/99 | 0    | 0.0                            | 52f7142ef606d8b1dec2ff3371c7452c8d7b81ea | 31 116         |
| generator://csmith-v0/99 | 0    | 0.268005818128586              | d8c05bd41b7a6c6157b6a8f0f5093907c7cc7ecf | 31 116 103     |
| generator://csmith-v0/99 | 0    | 0.288621664047241              | c4d7ecd3807793a0d8bc281104c7f5a8aa4670f9 | 31 116 103 109 |
  1. Observations: records pickled, compressed, and text observation values for each unique state.

Caveats of this implementation:

  1. Only LlvmEnv environments may be wrapped.
  2. The wrapped environment must have an observation space and reward space set.
  3. The observation spaces and reward spaces that are logged to database are hardcoded. To change what is recorded, you must copy and modify this implementation.
  4. Writing to the database is synchronous and adds significant overhead to the compute cost of the environment.

Fixes #678.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label May 12, 2022
@codecov-commenter
Copy link

codecov-commenter commented May 12, 2022

Codecov Report

Merging #679 (426f873) into development (947b567) will decrease coverage by 0.24%.
The diff coverage is 92.59%.

@@               Coverage Diff               @@
##           development     #679      +/-   ##
===============================================
- Coverage        88.61%   88.36%   -0.25%     
===============================================
  Files              126      127       +1     
  Lines             7597     7678      +81     
===============================================
+ Hits              6732     6785      +53     
- Misses             865      893      +28     
Impacted Files Coverage Δ
compiler_gym/util/timer.py 100.00% <ø> (ø)
compiler_gym/wrappers/sqlite_logger.py 92.40% <92.40%> (ø)
compiler_gym/wrappers/__init__.py 100.00% <100.00%> (ø)
compiler_gym/__init__.py 87.50% <0.00%> (-6.25%) ⬇️
...loop_tool/service/loop_tool_compilation_session.py 86.48% <0.00%> (-4.73%) ⬇️
compiler_gym/util/download.py 94.11% <0.00%> (-2.95%) ⬇️
compiler_gym/service/connection.py 78.66% <0.00%> (-2.00%) ⬇️
compiler_gym/envs/llvm/datasets/cbench.py 79.42% <0.00%> (-1.09%) ⬇️
compiler_gym/random_search.py 92.10% <0.00%> (-0.53%) ⬇️
...ompiler_gym/service/client_service_compiler_env.py 90.85% <0.00%> (-0.42%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 947b567...426f873. Read the comment docs.

@ChrisCummins ChrisCummins merged commit 51ff9d1 into facebookresearch:development May 16, 2022
@ChrisCummins ChrisCummins deleted the feature/db-wrapper branch May 16, 2022 20:44
@ChrisCummins ChrisCummins mentioned this pull request May 24, 2022
ChrisCummins added a commit that referenced this pull request May 24, 2022
This release adds a new compiler environment, new APIs, and a suite of backend
improvements to improve the flexibility of CompilerGym environments. Many thanks
to code contributors: @sogartar, @KyleHerndon, @SoumyajitKarmakar@uduse, and
@anthony0727!

Highlights of this release include:

- [mlir] Began work on a new environment for matrix multiplication using MLIR
  ([#652](#652), thanks
  @KyleHerndon and @sogartar!). Note this environment is not yet included in the
  pypi package and must be [compiled from
  source](https://github.com/facebookresearch/CompilerGym/blob/development/INSTALL.md#building-from-source-with-cmake).
- [llvm] Added a new `env.benchmark_from_clang_invocation()` method ([#577](#577)) that can be used for constructing LLVM environment automatically from C/C++ compiler invocations. This makes it much easier to integrate CompilerGym with your existing build scripts.
- Added three new wrapper classes: `Counter`, that provides op counts for
  analysis ([#683](#683));
  `SynchronousSqliteLogger`, that provides logging of environment interactions
  to a relational database
  ([#679](#679)), and
  `ForkOnStep` that provides an `undo()` operation
  ([#682](#682)).
- Added `reward_space` and `observation_space` parameters to `env.reset()`
  ([#659](#659), thanks
  @SoumyajitKarmakar!)

This release includes a number of improvements to the backend APIs that make it
easier to write new CompilerGym environments:

- Refactored the backend to make `CompilerEnv` an abstract interface, and
  `ClientServiceCompilerEnv` the concrete implementation of this interface. This
  enables new environments to be implemented without using gRPC
  ([#633](#633), thanks
  @sogartar!).
- Extended the support for different types of action and observation spaces
  ([#641](#641),
  [#643](#643), thanks
  @sogartar!), including new `Permutation` and `SpaceSequence` spaces
  ([#645](#645), thanks
  @sogartar!)..
- Added a new `disk/` subdirectory to compiler service's working directories,
  which is symlinked to an on-disk location for devices which support in-memory
  working directories. This fixes a bug with leftover temporary directories from
  LLVM ([#672](#672)).

This release also includes numerous bug fixes and improvements, many of which
were reported or fixed by the community. For example, fixing a bug in cache file
locations ([#656](#656),
thanks @uduse!), and a missing flag definition in example code
([#684](#684), thanks
@anthony0727!).

**Full Changelog**:
v0.2.3...v0.2.4

This release brings in deprecating changes to the core `env.step()` routine, and
lays the groundwork for enabling new types of compiler optimizations to be
exposed through CompilerGym. Many thanks to code contributors: @mostafaelhoushi,
@sogartar, @KyleHerndon, @uduse, @parthchadha, and @xtremey!

Highlights of this release include:

- Added a new `TextSizeInBytes` observation space for LLVM
  ([#575](#575)).
* Added a new PPO leaderboard entry
  ([#580](#580). Thanks
  @xtremey!
- Fixed a bug in which temporary directories created by the LLVM environment
  were not cleaned up
  ([#592](#592)).
- **[Backend]** The function `createAndRunCompilerGymService` now returns an
  int, which is the exit return code
  ([#592](#592)).
- Improvements to the examples documentation
  ([#548](#548)) and FAQ
  ([#586](#586))

Deprecations and breaking changes:

- `CompilerEnv.step` no longer accepts a list of actions
  ([#627](#627)). A new
  method, `CompilerEnv.multistep` provides this functionality. This is to
  provide compatibility with environments whose action spaces are lists. To
  update your code, replace any calls to `env.step()` which take a list of
  actions to use `env.multistep()`. Thanks @sogartar!
- The arguments `observations` and `rewards` to `step()` have been renamed
  `observation_spaces` and `reward_spaces`, respectively
  ([#627](#627)).
- `Reward.id` has been renamed `Reward.name`
  ([#565](#565),
  [#612](#612)). Thanks
  @parthchadha!
* The backend protocol buffer schema has been updated to natively support more
  types of observation and action, and to support nested spaces
  ([#531](#531)). Thanks
  @sogartar!
ChrisCummins added a commit that referenced this pull request May 24, 2022
This release adds a new compiler environment, new APIs, and a suite of backend
improvements to improve the flexibility of CompilerGym environments. Many thanks
to code contributors: @sogartar, @KyleHerndon, @SoumyajitKarmakar@uduse, and
@anthony0727!

Highlights of this release include:

- [mlir] Began work on a new environment for matrix multiplication using MLIR
  ([#652](#652), thanks
  @KyleHerndon and @sogartar!). Note this environment is not yet included in the
  pypi package and must be [compiled from
  source](https://github.com/facebookresearch/CompilerGym/blob/development/INSTALL.md#building-from-source-with-cmake).
- [llvm] Added a new `env.benchmark_from_clang_invocation()` method ([#577](#577)) that can be used for constructing LLVM environment automatically from C/C++ compiler invocations. This makes it much easier to integrate CompilerGym with your existing build scripts.
- Added three new wrapper classes: `Counter`, that provides op counts for
  analysis ([#683](#683));
  `SynchronousSqliteLogger`, that provides logging of environment interactions
  to a relational database
  ([#679](#679)), and
  `ForkOnStep` that provides an `undo()` operation
  ([#682](#682)).
- Added `reward_space` and `observation_space` parameters to `env.reset()`
  ([#659](#659), thanks
  @SoumyajitKarmakar!)

This release includes a number of improvements to the backend APIs that make it
easier to write new CompilerGym environments:

- Refactored the backend to make `CompilerEnv` an abstract interface, and
  `ClientServiceCompilerEnv` the concrete implementation of this interface. This
  enables new environments to be implemented without using gRPC
  ([#633](#633), thanks
  @sogartar!).
- Extended the support for different types of action and observation spaces
  ([#641](#641),
  [#643](#643), thanks
  @sogartar!), including new `Permutation` and `SpaceSequence` spaces
  ([#645](#645), thanks
  @sogartar!)..
- Added a new `disk/` subdirectory to compiler service's working directories,
  which is symlinked to an on-disk location for devices which support in-memory
  working directories. This fixes a bug with leftover temporary directories from
  LLVM ([#672](#672)).

This release also includes numerous bug fixes and improvements, many of which
were reported or fixed by the community. For example, fixing a bug in cache file
locations ([#656](#656),
thanks @uduse!), and a missing flag definition in example code
([#684](#684), thanks
@anthony0727!).

**Full Changelog**:
v0.2.3...v0.2.4

This release brings in deprecating changes to the core `env.step()` routine, and
lays the groundwork for enabling new types of compiler optimizations to be
exposed through CompilerGym. Many thanks to code contributors: @mostafaelhoushi,
@sogartar, @KyleHerndon, @uduse, @parthchadha, and @xtremey!

Highlights of this release include:

- Added a new `TextSizeInBytes` observation space for LLVM
  ([#575](#575)).
* Added a new PPO leaderboard entry
  ([#580](#580). Thanks
  @xtremey!
- Fixed a bug in which temporary directories created by the LLVM environment
  were not cleaned up
  ([#592](#592)).
- **[Backend]** The function `createAndRunCompilerGymService` now returns an
  int, which is the exit return code
  ([#592](#592)).
- Improvements to the examples documentation
  ([#548](#548)) and FAQ
  ([#586](#586))

Deprecations and breaking changes:

- `CompilerEnv.step` no longer accepts a list of actions
  ([#627](#627)). A new
  method, `CompilerEnv.multistep` provides this functionality. This is to
  provide compatibility with environments whose action spaces are lists. To
  update your code, replace any calls to `env.step()` which take a list of
  actions to use `env.multistep()`. Thanks @sogartar!
- The arguments `observations` and `rewards` to `step()` have been renamed
  `observation_spaces` and `reward_spaces`, respectively
  ([#627](#627)).
- `Reward.id` has been renamed `Reward.name`
  ([#565](#565),
  [#612](#612)). Thanks
  @parthchadha!
* The backend protocol buffer schema has been updated to natively support more
  types of observation and action, and to support nested spaces
  ([#531](#531)). Thanks
  @sogartar!
ChrisCummins added a commit that referenced this pull request May 24, 2022
This release adds a new compiler environment, new APIs, and a suite of backend
improvements to improve the flexibility of CompilerGym environments. Many thanks
to code contributors: @sogartar, @KyleHerndon, @SoumyajitKarmakar@uduse, and
@anthony0727!

Highlights of this release include:

- [mlir] Began work on a new environment for matrix multiplication using MLIR
  ([#652](#652), thanks
  @KyleHerndon and @sogartar!). Note this environment is not yet included in the
  pypi package and must be [compiled from
  source](https://github.com/facebookresearch/CompilerGym/blob/development/INSTALL.md#building-from-source-with-cmake).
- [llvm] Added a new `env.benchmark_from_clang_invocation()` method ([#577](#577)) that can be used for constructing LLVM environment automatically from C/C++ compiler invocations. This makes it much easier to integrate CompilerGym with your existing build scripts.
- Added three new wrapper classes: `Counter`, that provides op counts for
  analysis ([#683](#683));
  `SynchronousSqliteLogger`, that provides logging of environment interactions
  to a relational database
  ([#679](#679)), and
  `ForkOnStep` that provides an `undo()` operation
  ([#682](#682)).
- Added `reward_space` and `observation_space` parameters to `env.reset()`
  ([#659](#659), thanks
  @SoumyajitKarmakar!)

This release includes a number of improvements to the backend APIs that make it
easier to write new CompilerGym environments:

- Refactored the backend to make `CompilerEnv` an abstract interface, and
  `ClientServiceCompilerEnv` the concrete implementation of this interface. This
  enables new environments to be implemented without using gRPC
  ([#633](#633), thanks
  @sogartar!).
- Extended the support for different types of action and observation spaces
  ([#641](#641),
  [#643](#643), thanks
  @sogartar!), including new `Permutation` and `SpaceSequence` spaces
  ([#645](#645), thanks
  @sogartar!)..
- Added a new `disk/` subdirectory to compiler service's working directories,
  which is symlinked to an on-disk location for devices which support in-memory
  working directories. This fixes a bug with leftover temporary directories from
  LLVM ([#672](#672)).

This release also includes numerous bug fixes and improvements, many of which
were reported or fixed by the community. For example, fixing a bug in cache file
locations ([#656](#656),
thanks @uduse!), and a missing flag definition in example code
([#684](#684), thanks
@anthony0727!).

**Full Changelog**:
v0.2.3...v0.2.4

This release brings in deprecating changes to the core `env.step()` routine, and
lays the groundwork for enabling new types of compiler optimizations to be
exposed through CompilerGym. Many thanks to code contributors: @mostafaelhoushi,
@sogartar, @KyleHerndon, @uduse, @parthchadha, and @xtremey!

Highlights of this release include:

- Added a new `TextSizeInBytes` observation space for LLVM
  ([#575](#575)).
* Added a new PPO leaderboard entry
  ([#580](#580). Thanks
  @xtremey!
- Fixed a bug in which temporary directories created by the LLVM environment
  were not cleaned up
  ([#592](#592)).
- **[Backend]** The function `createAndRunCompilerGymService` now returns an
  int, which is the exit return code
  ([#592](#592)).
- Improvements to the examples documentation
  ([#548](#548)) and FAQ
  ([#586](#586))

Deprecations and breaking changes:

- `CompilerEnv.step` no longer accepts a list of actions
  ([#627](#627)). A new
  method, `CompilerEnv.multistep` provides this functionality. This is to
  provide compatibility with environments whose action spaces are lists. To
  update your code, replace any calls to `env.step()` which take a list of
  actions to use `env.multistep()`. Thanks @sogartar!
- The arguments `observations` and `rewards` to `step()` have been renamed
  `observation_spaces` and `reward_spaces`, respectively
  ([#627](#627)).
- `Reward.id` has been renamed `Reward.name`
  ([#565](#565),
  [#612](#612)). Thanks
  @parthchadha!
* The backend protocol buffer schema has been updated to natively support more
  types of observation and action, and to support nested spaces
  ([#531](#531)). Thanks
  @sogartar!
@ChrisCummins ChrisCummins mentioned this pull request May 25, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implementation of State Transition Dataset Wrapper
3 participants