Permalink
Browse files

Initial commit for public release

  • Loading branch information...
jma127 committed May 2, 2018
0 parents commit 113aba73ec0bc9d60bdb00b3c439bc60fecabc89
Showing with 32,811 additions and 0 deletions.
  1. +35 −0 .circleci/config.yml
  2. +34 −0 .circleci/docker/ubuntu_1804/Dockerfile
  3. +85 −0 .clang-format
  4. +8 −0 .flake8
  5. +28 −0 .gitignore
  6. +21 −0 .gitmodules
  7. +81 −0 CMakeLists.txt
  8. +7 −0 CODE_OF_CONDUCT.rst
  9. +63 −0 CONTRIBUTING.rst
  10. +30 −0 LICENSE
  11. +32 −0 Makefile
  12. +130 −0 README.rst
  13. +11 −0 codetools/autoformat_cpp_style.sh
  14. +11 −0 codetools/autoformat_py_style.sh
  15. +47 −0 codetools/check_cpp_style.sh
  16. +6 −0 codetools/check_py_style.sh
  17. +7 −0 codetools/run_all_autoformatters.sh
  18. +7 −0 codetools/run_all_static_checks.sh
  19. +108 −0 design_doc/ai_client.md
  20. +55 −0 design_doc/game_base.md
  21. +123 −0 design_doc/game_context.md
  22. +123 −0 design_doc/go_game.md
  23. +224 −0 design_doc/mcts.md
  24. +39 −0 design_doc/model_interface.md
  25. +35 −0 design_doc/options.md
  26. +32 −0 design_doc/sampler.md
  27. +51 −0 design_doc/trainer.md
  28. +4 −0 scripts/devmode_set_pythonpath.sh
  29. +372 −0 scripts/elfgames/go/console_lib.py
  30. +81 −0 scripts/elfgames/go/df_console.py
  31. +18 −0 scripts/elfgames/go/gtp.sh
  32. +204 −0 scripts/elfgames/go/selfplay.py
  33. +3 −0 scripts/elfgames/go/server_addrs.py
  34. +30 −0 scripts/elfgames/go/start_client.sh
  35. +57 −0 scripts/elfgames/go/start_selfplay.sh
  36. +32 −0 scripts/elfgames/go/start_server.sh
  37. +131 −0 scripts/elfgames/go/train.py
  38. +52 −0 src_cpp/elf/CMakeLists.txt
  39. +117 −0 src_cpp/elf/Pybind.cc
  40. +17 −0 src_cpp/elf/Pybind.h
  41. +119 −0 src_cpp/elf/ai/ai.h
  42. +165 −0 src_cpp/elf/ai/tree_search/mcts.h
  43. +523 −0 src_cpp/elf/ai/tree_search/tree_search.h
  44. +85 −0 src_cpp/elf/ai/tree_search/tree_search_alg.h
  45. +355 −0 src_cpp/elf/ai/tree_search/tree_search_base.h
  46. +573 −0 src_cpp/elf/ai/tree_search/tree_search_node.h
  47. +264 −0 src_cpp/elf/ai/tree_search/tree_search_options.h
  48. +103 −0 src_cpp/elf/base/common.h
  49. +485 −0 src_cpp/elf/base/context.h
  50. +352 −0 src_cpp/elf/base/ctrl.h
  51. +625 −0 src_cpp/elf/base/extractor.h
  52. +76 −0 src_cpp/elf/base/hist.h
  53. +299 −0 src_cpp/elf/base/sharedmem.h
  54. +189 −0 src_cpp/elf/comm/broadcast.h
  55. +418 −0 src_cpp/elf/comm/comm.h
  56. +69 −0 src_cpp/elf/comm/primitive.h
  57. +110 −0 src_cpp/elf/concurrency/ConcurrentQueue.h
  58. +20 −0 src_cpp/elf/concurrency/Counter.cc
  59. +153 −0 src_cpp/elf/concurrency/Counter.h
  60. +39 −0 src_cpp/elf/concurrency/TBBHashers.h
  61. +336 −0 src_cpp/elf/distributed/shared_reader.h
  62. +54 −0 src_cpp/elf/distributed/shared_replay_buffer.h
  63. +271 −0 src_cpp/elf/distributed/shared_rw_buffer2.h
  64. +313 −0 src_cpp/elf/distributed/zmq_util.h
  65. +5,157 −0 src_cpp/elf/legacy/pybind_helper.h
  66. +52 −0 src_cpp/elf/legacy/python_options_utils_cpp.h
  67. +36 −0 src_cpp/elf/logging/IndexedLoggerFactory.cc
  68. +72 −0 src_cpp/elf/logging/IndexedLoggerFactory.h
  69. +52 −0 src_cpp/elf/logging/Levels.cc
  70. +34 −0 src_cpp/elf/logging/Levels.h
  71. +68 −0 src_cpp/elf/logging/Pybind.cc
  72. +19 −0 src_cpp/elf/logging/Pybind.h
  73. +49 −0 src_cpp/elf/options/OptionMap.cc
  74. +110 −0 src_cpp/elf/options/OptionMap.h
  75. +93 −0 src_cpp/elf/options/OptionMapTest.cc
  76. +79 −0 src_cpp/elf/options/OptionSpec.cc
  77. +300 −0 src_cpp/elf/options/OptionSpec.h
  78. +101 −0 src_cpp/elf/options/OptionSpecTest.cc
  79. +36 −0 src_cpp/elf/options/OptionSpecTest.h
  80. +28 −0 src_cpp/elf/options/Pybind.cc
  81. +19 −0 src_cpp/elf/options/Pybind.h
  82. +15 −0 src_cpp/elf/pybind_module.cc
  83. +61 −0 src_cpp/elf/utils/json_utils.h
  84. +31 −0 src_cpp/elf/utils/member_check.h
  85. +213 −0 src_cpp/elf/utils/utils.h
  86. +57 −0 src_cpp/elfgames/go/CMakeLists.txt
  87. +60 −0 src_cpp/elfgames/go/Pybind.cc
  88. +19 −0 src_cpp/elfgames/go/Pybind.h
  89. +86 −0 src_cpp/elfgames/go/README.md
  90. +2,119 −0 src_cpp/elfgames/go/base/board.cc
  91. +458 −0 src_cpp/elfgames/go/base/board.h
  92. +286 −0 src_cpp/elfgames/go/base/board_feature.cc
  93. +173 −0 src_cpp/elfgames/go/base/board_feature.h
  94. +107 −0 src_cpp/elfgames/go/base/board_feature_test.cc
  95. +48 −0 src_cpp/elfgames/go/base/common.cc
  96. +66 −0 src_cpp/elfgames/go/base/common.h
  97. +93 −0 src_cpp/elfgames/go/base/coord_test.cc
  98. +15 −0 src_cpp/elfgames/go/base/go_common.h
  99. +143 −0 src_cpp/elfgames/go/base/go_state.cc
  100. +242 −0 src_cpp/elfgames/go/base/go_state.h
  101. +671 −0 src_cpp/elfgames/go/base/go_test.cc
  102. +160 −0 src_cpp/elfgames/go/base/hash_num.h
  103. +289 −0 src_cpp/elfgames/go/base/symmetry_test.cc
  104. +95 −0 src_cpp/elfgames/go/base/test_utils.h
  105. +73 −0 src_cpp/elfgames/go/client_manager.cc
  106. +252 −0 src_cpp/elfgames/go/client_manager.h
  107. +381 −0 src_cpp/elfgames/go/ctrl_eval.h
  108. +458 −0 src_cpp/elfgames/go/ctrl_selfplay.h
  109. +89 −0 src_cpp/elfgames/go/ctrl_utils.h
  110. +78 −0 src_cpp/elfgames/go/data_loader.h
  111. +324 −0 src_cpp/elfgames/go/fair_pick.h
  112. +60 −0 src_cpp/elfgames/go/game_base.h
  113. +326 −0 src_cpp/elfgames/go/game_context.h
  114. +669 −0 src_cpp/elfgames/go/game_ctrl.h
  115. +216 −0 src_cpp/elfgames/go/game_feature.h
  116. +425 −0 src_cpp/elfgames/go/game_selfplay.cc
  117. +83 −0 src_cpp/elfgames/go/game_selfplay.h
  118. +61 −0 src_cpp/elfgames/go/game_stats.h
  119. +58 −0 src_cpp/elfgames/go/game_train.cc
  120. +29 −0 src_cpp/elfgames/go/game_train.h
  121. +107 −0 src_cpp/elfgames/go/game_utils.h
  122. +263 −0 src_cpp/elfgames/go/go_game_specific.h
  123. +126 −0 src_cpp/elfgames/go/go_state_ext.cc
  124. +313 −0 src_cpp/elfgames/go/go_state_ext.h
  125. +54 −0 src_cpp/elfgames/go/mcts/ai.h
  126. +378 −0 src_cpp/elfgames/go/mcts/mcts.h
  127. +335 −0 src_cpp/elfgames/go/mcts/mcts_test.cc
  128. +15 −0 src_cpp/elfgames/go/pybind_module.cc
  129. +535 −0 src_cpp/elfgames/go/record.h
  130. +286 −0 src_cpp/elfgames/go/sgf/sgf.cc
  131. +289 −0 src_cpp/elfgames/go/sgf/sgf.h
  132. +158 −0 src_cpp/elfgames/go/sgf/sgf_test.cc
  133. +14 −0 src_py/elf/__init__.py
  134. +117 −0 src_py/elf/context_utils.py
  135. +10 −0 src_py/elf/logging/__init__.py
  136. +31 −0 src_py/elf/logging/configuration.py
  137. +27 −0 src_py/elf/more_labels.py
  138. +9 −0 src_py/elf/options/__init__.py
  139. +35 −0 src_py/elf/options/import_options.py
  140. +44 −0 src_py/elf/options/py_option_map.py
  141. +81 −0 src_py/elf/options/py_option_spec.py
  142. +463 −0 src_py/elf/utils_elf.py
  143. +76 −0 src_py/elf/zmq_util.py
  144. +87 −0 src_py/elfgames/go/df_model.py
  145. +129 −0 src_py/elfgames/go/df_model2.py
  146. +296 −0 src_py/elfgames/go/df_model3.py
  147. +434 −0 src_py/elfgames/go/game.py
  148. +92 −0 src_py/elfgames/go/mcts_prediction.py
  149. +67 −0 src_py/elfgames/go/multiple_prediction.py
  150. +54 −0 src_py/rlpytorch/README.md
  151. +16 −0 src_py/rlpytorch/__init__.py
  152. +13 −0 src_py/rlpytorch/methods/__init__.py
  153. +83 −0 src_py/rlpytorch/methods/actor_critic.py
  154. +56 −0 src_py/rlpytorch/methods/discounted_reward.py
  155. +178 −0 src_py/rlpytorch/methods/policy_gradient.py
  156. +83 −0 src_py/rlpytorch/methods/q_learning.py
  157. +93 −0 src_py/rlpytorch/methods/rnn_actor_critic.py
  158. +74 −0 src_py/rlpytorch/methods/utils.py
  159. +74 −0 src_py/rlpytorch/methods/value_matcher.py
  160. +188 −0 src_py/rlpytorch/model_base.py
  161. +254 −0 src_py/rlpytorch/model_interface.py
  162. +289 −0 src_py/rlpytorch/model_loader.py
  163. +9 −0 src_py/rlpytorch/runner/__init__.py
  164. +98 −0 src_py/rlpytorch/runner/eval_iters.py
  165. +112 −0 src_py/rlpytorch/runner/multi_process.py
  166. +269 −0 src_py/rlpytorch/runner/parameter_server.py
  167. +155 −0 src_py/rlpytorch/runner/single_process.py
  168. +8 −0 src_py/rlpytorch/sampler/__init__.py
  169. +156 −0 src_py/rlpytorch/sampler/sample_methods.py
  170. +58 −0 src_py/rlpytorch/sampler/sampler.py
  171. +7 −0 src_py/rlpytorch/stats/__init__.py
  172. +230 −0 src_py/rlpytorch/stats/stats.py
  173. +9 −0 src_py/rlpytorch/trainer/__init__.py
  174. +132 −0 src_py/rlpytorch/trainer/lstm_trainer.py
  175. +60 −0 src_py/rlpytorch/trainer/timer.py
  176. +288 −0 src_py/rlpytorch/trainer/trainer.py
  177. +180 −0 src_py/rlpytorch/trainer/utils.py
  178. +7 −0 src_py/rlpytorch/utils/__init__.py
  179. +56 −0 src_py/rlpytorch/utils/fp16_utils.py
  180. +79 −0 src_py/rlpytorch/utils/hist_states.py
  181. +71 −0 src_py/rlpytorch/utils/size_utils.py
  182. +502 −0 src_py/rlpytorch/utils/utils.py
  183. +51 −0 third_party/CMakeLists.txt
  184. +1 −0 third_party/concurrentqueue
  185. +1 −0 third_party/cppzmq
  186. +1 −0 third_party/googletest
  187. +1 −0 third_party/json
  188. +1 −0 third_party/pybind11
  189. +1 −0 third_party/spdlog
  190. +1 −0 third_party/tbb
@@ -0,0 +1,35 @@
version: 2

jobs:
build_and_test_1804:
resource_class: xlarge
docker:
- image: pytorch/elf:ci_ubuntu_1804
steps:
- checkout
- run:
name: submodules
command: |
git submodule sync
git submodule update --init --recursive
- run:
name: cpp_style
command: ./codetools/check_cpp_style.sh
- run:
name: python_style
command: ./codetools/check_py_style.sh
- run:
name: compile
command: make clean all VERBOSE=1
- run:
name: cpp_tests
command: make test_cpp
- run:
name: python_tests
command: "true"

workflows:
version: 2
build_and_test_1804:
jobs:
- build_and_test_1804
@@ -0,0 +1,34 @@
FROM ubuntu:bionic

WORKDIR /tmp/docker_init
SHELL ["/bin/bash", "-c"]

RUN apt-get update
RUN apt-get install -y \
clang-format \
clang-tidy \
cmake \
curl \
g++ \
gcc \
git \
libboost-all-dev \
libzmq3-dev \
;
RUN ln -sf `which clang-format-6.0` /usr/bin/clang-format
RUN curl https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh > conda_install.sh
RUN bash conda_install.sh -p /conda -b
ENV PATH=/conda/bin:${PATH}
RUN conda update -n base conda
RUN conda create -yn elf_env python=3.6
RUN source activate elf_env
RUN conda install -y \
flake8 \
numpy \
pytest \
zeromq \
;
RUN conda install -c conda-forge autopep8
RUN conda install -c pytorch pytorch torchvision cuda90

RUN rm -rf /tmp/docker_init
@@ -0,0 +1,85 @@
---
AccessModifierOffset: -1
AlignAfterOpenBracket: AlwaysBreak
AlignConsecutiveAssignments: false
AlignConsecutiveDeclarations: false
AlignEscapedNewlinesLeft: true
AlignOperands: false
AlignTrailingComments: false
AllowAllParametersOfDeclarationOnNextLine: false
AllowShortBlocksOnASingleLine: false
AllowShortCaseLabelsOnASingleLine: false
AllowShortFunctionsOnASingleLine: Empty
AllowShortIfStatementsOnASingleLine: false
AllowShortLoopsOnASingleLine: false
AlwaysBreakAfterReturnType: None
AlwaysBreakBeforeMultilineStrings: true
AlwaysBreakTemplateDeclarations: true
BinPackArguments: false
BinPackParameters: false
BraceWrapping:
AfterClass: false
AfterControlStatement: false
AfterEnum: false
AfterFunction: false
AfterNamespace: false
AfterObjCDeclaration: false
AfterStruct: false
AfterUnion: false
BeforeCatch: false
BeforeElse: false
IndentBraces: false
BreakBeforeBinaryOperators: None
BreakBeforeBraces: Attach
BreakBeforeTernaryOperators: true
BreakConstructorInitializersBeforeComma: false
ColumnLimit: 80
CommentPragmas: '^ IWYU pragma:'
ConstructorInitializerAllOnOneLineOrOnePerLine: true
ConstructorInitializerIndentWidth: 4
ContinuationIndentWidth: 4
Cpp11BracedListStyle: true
DerivePointerAlignment: false
DisableFormat: false
ForEachMacros: [ FOR_EACH_RANGE, FOR_EACH, ]
IncludeCategories:
- Regex: '^<.*\.h(pp)?>'
Priority: 1
- Regex: '^<.*'
Priority: 2
- Regex: '.*'
Priority: 3
IndentCaseLabels: true
IndentWidth: 2
IndentWrappedFunctionNames: false
KeepEmptyLinesAtTheStartOfBlocks: false
MacroBlockBegin: ''
MacroBlockEnd: ''
MaxEmptyLinesToKeep: 1
NamespaceIndentation: None
ObjCBlockIndentWidth: 2
ObjCSpaceAfterProperty: false
ObjCSpaceBeforeProtocolList: false
PenaltyBreakBeforeFirstCallParameter: 1
PenaltyBreakComment: 300
PenaltyBreakFirstLessLess: 120
PenaltyBreakString: 1000
PenaltyExcessCharacter: 1000000
PenaltyReturnTypeOnItsOwnLine: 200
PointerAlignment: Left
ReflowComments: true
SortIncludes: true
SpaceAfterCStyleCast: false
SpaceBeforeAssignmentOperators: true
SpaceBeforeParens: ControlStatements
SpaceInEmptyParentheses: false
SpacesBeforeTrailingComments: 1
SpacesInAngles: false
SpacesInContainerLiterals: true
SpacesInCStyleCastParentheses: false
SpacesInParentheses: false
SpacesInSquareBrackets: false
Standard: Cpp11
TabWidth: 8
UseTab: Never
...
@@ -0,0 +1,8 @@
[flake8]
exclude =
__init__.py
__pycache__,
.git,
build,
experimental,
third_party,
@@ -0,0 +1,28 @@
# Build and interpreter artifacts
*.a
*.bin
*.dSYM
*.o
*.pyc
*.so
build/
__pycache__/

# Random junk
*.sw?
*.db
*.json
*.sgf
*.swo
*.tree
.codemod.bookmark
.nfs*
tmp*.sh
tags
build/
save_dir*/
test-*/
Testing/
pachi-*/
experimental/patterns.prob
experimental/patterns.spat
@@ -0,0 +1,21 @@
[submodule "third_party/concurrentqueue"]
path = third_party/concurrentqueue
url = https://github.com/cameron314/concurrentqueue.git
[submodule "third_party/cppzmq"]
path = third_party/cppzmq
url = https://github.com/zeromq/cppzmq.git
[submodule "third_party/googletest"]
path = third_party/googletest
url = https://github.com/google/googletest.git
[submodule "third_party/json"]
path = third_party/json
url = https://github.com/nlohmann/json.git
[submodule "third_party/pybind11"]
path = third_party/pybind11
url = https://github.com/pybind/pybind11.git
[submodule "third_party/spdlog"]
path = third_party/spdlog
url = https://github.com/gabime/spdlog.git
[submodule "third_party/tbb"]
path = third_party/tbb
url = https://github.com/01org/tbb.git
@@ -0,0 +1,81 @@
cmake_minimum_required(VERSION 3.3)

# Basic third-party dependencies

find_package(Threads)

#find_package(Boost REQUIRED COMPONENTS
# thread
#)
#include_directories(${Boost_INCLUDE_DIR})

# Include third-party dependencies

add_subdirectory(
third_party
${CMAKE_CURRENT_BINARY_DIR}/third_party)
find_package(TBB REQUIRED tbb) # Need to find_package in parent scope

# Sanity checks

if("${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU")
# require at least gcc 7.1
if (CMAKE_CXX_COMPILER_VERSION VERSION_LESS 7.1)
message(FATAL_ERROR "GCC version must be at least 7.1!")
endif()
endif()

# Variables from Git

execute_process(COMMAND git rev-parse HEAD
OUTPUT_VARIABLE GIT_COMMIT_HASH)
string(STRIP ${GIT_COMMIT_HASH} GIT_COMMIT_HASH)
execute_process(COMMAND git diff-index --quiet HEAD --
RESULT_VARIABLE GIT_UNSTAGED)
if(${GIT_UNSTAGED})
set(GIT_STAGED_STRING unstaged)
else()
set(GIT_STAGED_STRING staged)
endif()

# Global compiler config

set(CMAKE_CXX_STANDARD 17)
set(PYBIND11_CPP_STANDARD -std=c++17)
set(CMAKE_CXX_FLAGS
"${CMAKE_CXX_FLAGS} -Wall -Werror -Wextra -Wno-register -fPIC -march=native")
set(CMAKE_CXX_FLAGS_RELEASE "-O3")
if(NOT CMAKE_BUILD_TYPE)
set(CMAKE_BUILD_TYPE Release)
endif()

# Define a convenience function for tests
function(add_cpp_tests prefix lib_to_link)
set(test_list ${ARGV})
list(REMOVE_AT test_list 0)
list(REMOVE_AT test_list 0)
foreach(test_file ${test_list})
string(REPLACE "/" "_" test_name ${test_file})
string(REPLACE ".cc" "" test_name ${test_name})
string(CONCAT test_name ${prefix} ${test_name})
add_executable(${test_name} ${test_file})
add_test(${test_name} ${test_name})
target_link_libraries(${test_name} ${lib_to_link} gtest)
endforeach(test_file)
endfunction(add_cpp_tests)

# Include everything in src_cpp

include_directories(${CMAKE_CURRENT_SOURCE_DIR}/src_cpp/)

# Main ELF library

add_subdirectory(
src_cpp/elf
${CMAKE_CURRENT_BINARY_DIR}/elf)

# ELF games

add_subdirectory(
src_cpp/elfgames/go
${CMAKE_CURRENT_BINARY_DIR}/elfgames/go)
@@ -0,0 +1,7 @@
===============
Code of Conduct
===============

Facebook has adopted a Code of Conduct that we expect project participants to adhere to. Please read the `full text`__ so that you can understand what actions will and will not be tolerated.

__ https://code.facebook.com/pages/876921332402685/open-source-code-of-conduct)
@@ -0,0 +1,63 @@
===================
Contributing to ELF
===================

We want to make contributing to this project as easy and transparent as possible.

Dependencies
============

In addition to the project dependencies, you'll need the following if you'd like to contribute code to ELF2::

sudo apt-get install clang-format clang-tidy
conda install flake8 pytest
conda install -c conda-forge autopep8

Pull Requests
=============

We actively welcome your pull requests.

1. Fork the repo and create your branch from ``master``.
2. If you've added code that should be tested, add tests.
3. If you've changed APIs, update the documentation.
4. Ensure the test suite passes.
5. Make sure your code lints (see the ``codetools/`` directory).
6. If you haven't already, complete the Contributor License Agreement ("CLA").

We have a continuous integration system that will double-check steps 4 and 5 for you.

Contributor License Agreement ("CLA")
=====================================

In order to accept your pull request, we need you to submit a CLA. You only need
to do this once to work on any of Facebook's open source projects.

Complete your CLA `here`__.

__ https://code.facebook.com/cla

Issues
======

We use GitHub issues to track public bugs. Please ensure your description is
clear and has sufficient instructions to be able to reproduce the issue.

This project is still an extremely unstable prototype so we will be limited in
the amount of support we're able to provide.

Coding Style
============

- For C++, we use a style very similar to the `HHVM style guide`__. This prescription is not formal, and our ``.clang-format`` file and existing code should eventually be a good source of truth. Due to a quirk of ``Python.h``, all ``pybind`` includes must be first in the include order.
- For Python, we use pep8.

__ https://github.com/facebook/hhvm/blob/master/hphp/doc/coding-conventions.md

We have tools to check your C++ and Python code in the ``codetools/`` directory

License
=======

By contributing to ELF, you agree that your contributions will be licensed
under the LICENSE file in the root directory of this source tree.
Oops, something went wrong.

0 comments on commit 113aba7

Please sign in to comment.