Support multiple GPUs #160

LWisteria · 2019-01-26T10:51:59Z

There is no more passed test by this PR because modification by #140 is just needed to pass multiple gpu tests. It provides only different GPU id but works on only same (primary) GPU device.

This PR works really on multiple GPUs. For example, performance of Chainer's mnist example is as follows:

$ python -m clpy train_mnist.py -g 0
epoch       main/loss   validation/main/loss  main/accuracy  validation/main/accuracy  elapsed_time
20          0.00664797  0.123965              0.998099       0.9805                    241.777

$ python -m clpy train_mnist_data_parallel.py
epoch       main/loss   validation/main/loss  main/accuracy  validation/main/accuracy  elapsed_time
20          0.0118985   0.0924893             0.996067       0.9775                    208.808

The result shows performance on multiple GPUs is faster than one GPU, evidence of multiple GPUs working.

LWisteria · 2019-01-26T10:53:16Z

Remains:

Add acknowledgement for environment provider
CI

jenkins-maekawa

Test (commit c604e8e) failed in titanv.

============================= test session starts ==============================
platform linux -- Python 3.6.5, pytest-4.1.1, py-1.7.0, pluggy-0.8.1
rootdir: /home/clpy-jenkins-slave/workspace/clpy_testing_titan, inifile: setup.cfg
collected 91 items

test_api.py .                                                            [  1%]
test_atomicAdd.py ..                                                     [  3%]
test_carray.py F                                                         [  4%]
test_clblast.py ...FFF................                                   [ 28%]
test_concatenate.py ........                                             [ 37%]
test_elementwise.py .                                                    [ 38%]
test_linalg.py ...                                                       [ 41%]
test_memory.py ........FFFFF                                             [ 56%]
test_ndarray.py ....................                                     [ 78%]
test_reduction.py ..........FFFF                                         [ 93%]
test_rollaxis.py ...                                                     [ 96%]
test_sample_rand.py ...                                                  [100%]

=================================== FAILURES ===================================
_________________ TestCArraywithChunk.test_get_CArrayIndexI_2 __________________

self = <test_carray.TestCArraywithChunk testMethod=test_get_CArrayIndexI_2>

    def setUp(self):
        # create chunk and free to prepare chunk in pool
        self.pool = clpy.backend.memory.SingleDeviceMemoryPool()
        clpy.backend.memory.set_allocator(self.pool.malloc)
>       self.pooled_chunk_size = clpy.backend.memory.subbuffer_alignment * 2
E       AttributeError: module 'clpy.backend.memory' has no attribute 'subbuffer_alignment'

test_carray.py:15: AttributeError
_______________________ TestBlas3GEMM.test_chunk_gemm_A ________________________

self = <test_clblast.TestBlas3GEMM testMethod=test_chunk_gemm_A>
dtype = <class 'numpy.float32'>, blasfunc = <built-in function sgemm>

    @for_each_dtype_and_blasfunc_pair(GEMM_pairs)
    def test_chunk_gemm_A(self, dtype, blasfunc):
        # create chunk and free to prepare chunk in pool
        pool = clpy.backend.memory.SingleDeviceMemoryPool()
        clpy.backend.memory.set_allocator(pool.malloc)
>       pooled_chunk_size = clpy.backend.memory.subbuffer_alignment * 2
E       AttributeError: module 'clpy.backend.memory' has no attribute 'subbuffer_alignment'

test_clblast.py:714: AttributeError
----------------------------- Captured stdout call -----------------------------
dtype: float32 , blasfunc: <built-in function sgemm>
_______________________ TestBlas3GEMM.test_chunk_gemm_B ________________________

self = <test_clblast.TestBlas3GEMM testMethod=test_chunk_gemm_B>

... and more 219 lines

jenkins-maekawa

Test (commit c604e8e) failed in vega.

============================= test session starts ==============================
platform linux -- Python 3.6.5, pytest-4.1.1, py-1.7.0, pluggy-0.8.1
rootdir: /home/clpy-jenkins-slave/workspace/clpy_testing_vega, inifile: setup.cfg
collected 91 items

test_api.py .                                                            [  1%]
test_atomicAdd.py ..                                                     [  3%]
test_carray.py F                                                         [  4%]
test_clblast.py ...FFF................                                   [ 28%]
test_concatenate.py ........                                             [ 37%]
test_elementwise.py .                                                    [ 38%]
test_linalg.py ...                                                       [ 41%]
test_memory.py ........FFFFF                                             [ 56%]
test_ndarray.py ....................                                     [ 78%]
test_reduction.py ..........FFFF                                         [ 93%]
test_rollaxis.py ...                                                     [ 96%]
test_sample_rand.py ...                                                  [100%]

=================================== FAILURES ===================================
_________________ TestCArraywithChunk.test_get_CArrayIndexI_2 __________________

self = <test_carray.TestCArraywithChunk testMethod=test_get_CArrayIndexI_2>

    def setUp(self):
        # create chunk and free to prepare chunk in pool
        self.pool = clpy.backend.memory.SingleDeviceMemoryPool()
        clpy.backend.memory.set_allocator(self.pool.malloc)
>       self.pooled_chunk_size = clpy.backend.memory.subbuffer_alignment * 2
E       AttributeError: module 'clpy.backend.memory' has no attribute 'subbuffer_alignment'

test_carray.py:15: AttributeError
_______________________ TestBlas3GEMM.test_chunk_gemm_A ________________________

self = <test_clblast.TestBlas3GEMM testMethod=test_chunk_gemm_A>
dtype = <class 'numpy.float32'>, blasfunc = <built-in function sgemm>

    @for_each_dtype_and_blasfunc_pair(GEMM_pairs)
    def test_chunk_gemm_A(self, dtype, blasfunc):
        # create chunk and free to prepare chunk in pool
        pool = clpy.backend.memory.SingleDeviceMemoryPool()
        clpy.backend.memory.set_allocator(pool.malloc)
>       pooled_chunk_size = clpy.backend.memory.subbuffer_alignment * 2
E       AttributeError: module 'clpy.backend.memory' has no attribute 'subbuffer_alignment'

test_clblast.py:714: AttributeError
----------------------------- Captured stdout call -----------------------------
dtype: float32 , blasfunc: <built-in function sgemm>
_______________________ TestBlas3GEMM.test_chunk_gemm_B ________________________

self = <test_clblast.TestBlas3GEMM testMethod=test_chunk_gemm_B>

... and more 219 lines

jenkins-maekawa

Test (commit cff3ef3) passed in vega.

jenkins-maekawa

Test (commit cff3ef3) passed in titanv.

LWisteria · 2019-01-26T11:25:28Z

CI passed. WIP will be removed after adding acknowledgement for the machine

jenkins-maekawa

Test (commit 85dd4bd) passed in vega.

jenkins-maekawa

Test (commit 85dd4bd) passed in titanv.

LWisteria · 2019-01-28T05:17:31Z

@neko-suki @ykitta-fixstars I am glad if either one of you review and merge this PR. If you could do it, please assign you and comment about when you will have done.

neko-suki · 2019-01-28T05:33:22Z

@LWisteria I will review.

neko-suki

@LWisteria I reviewed. Could you check my comment?

clpy/backend/opencl/utility.pyx

jenkins-maekawa

Test (commit ad3ca62) failed in vega.

clang++ -std=c++11 -D_GLIBCXX_USE_CXX11_ABI=1 -Wall -Wextra -O3 -pedantic-errors ultima.cpp -lclangTooling -lclangFrontendTool -lclangFrontend -lclangDriver -lclangSerialization -lclangCodeGen -lclangParse -lclangSema -lclangStaticAnalyzerFrontend -lclangStaticAnalyzerCheckers -lclangStaticAnalyzerCore -lclangAnalysis -lclangARCMigrate -lclangRewrite -lclangEdit -lclangAST -lclangLex -lclangBasic -lclang `llvm-config --libs --system-libs` -fno-rtti -o ultima
clang++ -std=c++11 -D_GLIBCXX_USE_CXX11_ABI=1 -Wall -Wextra -O3 -pedantic-errors headercvt.cpp -lclangTooling -lclangFrontendTool -lclangFrontend -lclangDriver -lclangSerialization -lclangCodeGen -lclangParse -lclangSema -lclangStaticAnalyzerFrontend -lclangStaticAnalyzerCheckers -lclangStaticAnalyzerCore -lclangAnalysis -lclangEdit -lclangAST -lclangLex -lclangBasic -lclang `llvm-config --libs --system-libs` -fno-rtti -o headercvt
./headercvt stub.c -- -I /usr/local/lib/clang/6.0.0/include
cp -f func_decl.pxi ../clpy/backend/opencl/func_decl.pxi
cp -f preprocessor_defines.pxi ../clpy/backend/opencl/preprocessor_defines.pxi
cp -f types.pxi ../clpy/backend/opencl/types.pxi
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
building ultima started
building headercvt
launching headercvt (converting cl.h)...
Options: {'profile': False, 'linetrace': False, 'annotate': False}
Include directories: []
Library directories: []
running develop
running egg_info
creating clpy.egg-info
writing clpy.egg-info/PKG-INFO
writing dependency_links to clpy.egg-info/dependency_links.txt
writing requirements to clpy.egg-info/requires.txt
writing top-level names to clpy.egg-info/top_level.txt
writing manifest file 'clpy.egg-info/SOURCES.txt'
reading manifest file 'clpy.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'clpy.egg-info/SOURCES.txt'
running build_ext
Include directories: []
Library directories: []
creating /tmp/tmpsgulyd0k/tmp
creating /tmp/tmpsgulyd0k/tmp/tmpsgulyd0k
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -c /tmp/tmpsgulyd0k/a.cpp -o /tmp/tmpsgulyd0k/tmp/tmpsgulyd0k/a.o
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
g++ -pthread -shared -L/home/clpy-jenkins-slave/.pyenv/versions/3.6.5/lib -L/home/clpy-jenkins-slave/.pyenv/versions/3.6.5/lib /tmp/tmpsgulyd0k/tmp/tmpsgulyd0k/a.o -o /tmp/tmpsgulyd0k/liba.so
creating /tmp/tmpjwjz5gdi/tmp
creating /tmp/tmpjwjz5gdi/tmp/tmpjwjz5gdi
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -c /tmp/tmpjwjz5gdi/a.cpp -o /tmp/tmpjwjz5gdi/tmp/tmpjwjz5gdi/a.o
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
g++ -pthread -shared -L/home/clpy-jenkins-slave/.pyenv/versions/3.6.5/lib -L/home/clpy-jenkins-slave/.pyenv/versions/3.6.5/lib /tmp/tmpjwjz5gdi/tmp/tmpjwjz5gdi/a.o -lOpenCL -lclblast -o /tmp/tmpjwjz5gdi/liba.so
/home/clpy-jenkins-slave/workspace/clpy_testing_vega/venv/lib/python3.6/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /home/clpy-jenkins-slave/workspace/clpy_testing_vega/clpy/backend/device.pxd
  tree = Parsing.p_module(s, pxd, full_module_name)
/home/clpy-jenkins-slave/workspace/clpy_testing_vega/venv/lib/python3.6/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /home/clpy-jenkins-slave/workspace/clpy_testing_vega/clpy/backend/function.pxd
  tree = Parsing.p_module(s, pxd, full_module_name)
warning: clpy/backend/function.pyx:220:23: Unreachable code
warning: clpy/backend/function.pyx:230:8: Unreachable code
/home/clpy-jenkins-slave/workspace/clpy_testing_vega/venv/lib/python3.6/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /home/clpy-jenkins-slave/workspace/clpy_testing_vega/clpy/backend/memory.pxd
  tree = Parsing.p_module(s, pxd, full_module_name)
warning: clpy/backend/memory.pyx:44:37: Unreachable code
/home/clpy-jenkins-slave/workspace/clpy_testing_vega/venv/lib/python3.6/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /home/clpy-jenkins-slave/workspace/clpy_testing_vega/clpy/backend/opencl/api.pxd
  tree = Parsing.p_module(s, pxd, full_module_name)
/home/clpy-jenkins-slave/workspace/clpy_testing_vega/venv/lib/python3.6/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /home/clpy-jenkins-slave/workspace/clpy_testing_vega/clpy/backend/opencl/clblast/clblast.pxd

... and more 71 lines

jenkins-maekawa

Test (commit ad3ca62) failed in titanv.

clang++ -std=c++11 -D_GLIBCXX_USE_CXX11_ABI=1 -Wall -Wextra -O3 -pedantic-errors ultima.cpp -lclangTooling -lclangFrontendTool -lclangFrontend -lclangDriver -lclangSerialization -lclangCodeGen -lclangParse -lclangSema -lclangStaticAnalyzerFrontend -lclangStaticAnalyzerCheckers -lclangStaticAnalyzerCore -lclangAnalysis -lclangARCMigrate -lclangRewrite -lclangEdit -lclangAST -lclangLex -lclangBasic -lclang `llvm-config --libs --system-libs` -fno-rtti -o ultima
clang++ -std=c++11 -D_GLIBCXX_USE_CXX11_ABI=1 -Wall -Wextra -O3 -pedantic-errors headercvt.cpp -lclangTooling -lclangFrontendTool -lclangFrontend -lclangDriver -lclangSerialization -lclangCodeGen -lclangParse -lclangSema -lclangStaticAnalyzerFrontend -lclangStaticAnalyzerCheckers -lclangStaticAnalyzerCore -lclangAnalysis -lclangEdit -lclangAST -lclangLex -lclangBasic -lclang `llvm-config --libs --system-libs` -fno-rtti -o headercvt
./headercvt stub.c -- -I /usr/local/lib/clang/6.0.0/include -I/usr/local/cuda/include
cp -f func_decl.pxi ../clpy/backend/opencl/func_decl.pxi
cp -f preprocessor_defines.pxi ../clpy/backend/opencl/preprocessor_defines.pxi
cp -f types.pxi ../clpy/backend/opencl/types.pxi
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
building ultima started
building headercvt
launching headercvt (converting cl.h)...
Options: {'profile': False, 'linetrace': False, 'annotate': False}
Include directories: ['/usr/local/cuda/include']
Library directories: ['/usr/local/cuda/lib64']
running develop
running egg_info
creating clpy.egg-info
writing clpy.egg-info/PKG-INFO
writing dependency_links to clpy.egg-info/dependency_links.txt
writing requirements to clpy.egg-info/requires.txt
writing top-level names to clpy.egg-info/top_level.txt
writing manifest file 'clpy.egg-info/SOURCES.txt'
reading manifest file 'clpy.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'clpy.egg-info/SOURCES.txt'
running build_ext
Include directories: ['/usr/local/cuda/include']
Library directories: ['/usr/local/cuda/lib64']
creating /tmp/tmpy_ncs1qt/tmp
creating /tmp/tmpy_ncs1qt/tmp/tmpy_ncs1qt
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/usr/local/cuda/include -c /tmp/tmpy_ncs1qt/a.cpp -o /tmp/tmpy_ncs1qt/tmp/tmpy_ncs1qt/a.o
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
g++ -pthread -shared -L/home/clpy-jenkins-slave/.pyenv/versions/3.6.5/lib -L/home/clpy-jenkins-slave/.pyenv/versions/3.6.5/lib /tmp/tmpy_ncs1qt/tmp/tmpy_ncs1qt/a.o -o /tmp/tmpy_ncs1qt/liba.so
creating /tmp/tmpjnkz1hdy/tmp
creating /tmp/tmpjnkz1hdy/tmp/tmpjnkz1hdy
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -c /tmp/tmpjnkz1hdy/a.cpp -o /tmp/tmpjnkz1hdy/tmp/tmpjnkz1hdy/a.o
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
g++ -pthread -shared -L/home/clpy-jenkins-slave/.pyenv/versions/3.6.5/lib -L/home/clpy-jenkins-slave/.pyenv/versions/3.6.5/lib /tmp/tmpjnkz1hdy/tmp/tmpjnkz1hdy/a.o -L/usr/local/cuda/lib64 -lOpenCL -lclblast -o /tmp/tmpjnkz1hdy/liba.so
/home/clpy-jenkins-slave/workspace/clpy_testing_titan/venv/lib/python3.6/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /home/clpy-jenkins-slave/workspace/clpy_testing_titan/clpy/backend/device.pxd
  tree = Parsing.p_module(s, pxd, full_module_name)
/home/clpy-jenkins-slave/workspace/clpy_testing_titan/venv/lib/python3.6/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /home/clpy-jenkins-slave/workspace/clpy_testing_titan/clpy/backend/function.pxd
  tree = Parsing.p_module(s, pxd, full_module_name)
warning: clpy/backend/function.pyx:220:23: Unreachable code
warning: clpy/backend/function.pyx:230:8: Unreachable code
/home/clpy-jenkins-slave/workspace/clpy_testing_titan/venv/lib/python3.6/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /home/clpy-jenkins-slave/workspace/clpy_testing_titan/clpy/backend/memory.pxd
  tree = Parsing.p_module(s, pxd, full_module_name)
warning: clpy/backend/memory.pyx:44:37: Unreachable code
/home/clpy-jenkins-slave/workspace/clpy_testing_titan/venv/lib/python3.6/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /home/clpy-jenkins-slave/workspace/clpy_testing_titan/clpy/backend/opencl/api.pxd
  tree = Parsing.p_module(s, pxd, full_module_name)
/home/clpy-jenkins-slave/workspace/clpy_testing_titan/venv/lib/python3.6/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /home/clpy-jenkins-slave/workspace/clpy_testing_titan/clpy/backend/opencl/clblast/clblast.pxd

... and more 71 lines

jenkins-maekawa

Test (commit ddb9a1f) passed in titanv.

jenkins-maekawa

Test (commit ddb9a1f) passed in vega.

neko-suki · 2019-01-31T01:06:57Z

LGTM

LWisteria added 11 commits January 25, 2019 17:38

Get all devices

af5fc6e

Check all devices' version

794e825

Get memory alignment when it's needed

1456a62

Rename get_devices_ptrs -> get_devices

c225d9a

-primary_device variavle

ad48e35

Create all command queues correspanding to all devices

fec9d85

Multiple typeof_size_t

22de76f

Move current_device_id into env

5aaf2ae

+return type of GetProgramBuildLog

1209c97

Get build log on all devices

117717d

Return cl_device/queue etc correspanding to current id

a7e48eb

LWisteria self-assigned this Jan 26, 2019

LWisteria changed the title ~~Support multiple GPUs~~ WIP: Support multiple GPUs Jan 26, 2019

jenkins-maekawa suggested changes Jan 26, 2019

View reviewed changes

LWisteria added 2 commits January 26, 2019 20:09

subbuffer_alignment->_allocation_unit_size on tests

85b9762

flake8

df6c2c9

jenkins-maekawa approved these changes Jan 26, 2019

View reviewed changes

+Acknowledgment

5fddcf2

LWisteria changed the title ~~WIP: Support multiple GPUs~~ Support multiple GPUs Jan 28, 2019

jenkins-maekawa approved these changes Jan 28, 2019

View reviewed changes

LWisteria removed their assignment Jan 28, 2019

neko-suki self-assigned this Jan 28, 2019

neko-suki self-requested a review January 28, 2019 05:32

neko-suki reviewed Jan 31, 2019

View reviewed changes

clpy/backend/opencl/utility.pyx Outdated Show resolved Hide resolved

fix typo: typess->types

01a18dc

jenkins-maekawa suggested changes Jan 31, 2019

View reviewed changes

fix 01a18dc

9a1b4e1

jenkins-maekawa approved these changes Jan 31, 2019

View reviewed changes

neko-suki approved these changes Jan 31, 2019

View reviewed changes

jenkins-maekawa approved these changes Jan 31, 2019

View reviewed changes

neko-suki merged commit d4d4c8c into clpy Jan 31, 2019

This was referenced Jan 31, 2019

Support multiple devices #4

Closed

cl_device_type is not consistent #11

Closed

LWisteria mentioned this pull request Feb 11, 2019

linear.py fails with CLBlastInvalidValue exception #171

Closed

LWisteria deleted the 4-multi_gpu branch February 14, 2019 03:44

LWisteria mentioned this pull request Feb 18, 2019

Multiple contexts for multiple devices #175

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support multiple GPUs #160

Support multiple GPUs #160

LWisteria commented Jan 26, 2019 •

edited

LWisteria commented Jan 26, 2019

jenkins-maekawa left a comment

jenkins-maekawa left a comment

jenkins-maekawa left a comment

jenkins-maekawa left a comment

LWisteria commented Jan 26, 2019 •

edited

jenkins-maekawa left a comment

jenkins-maekawa left a comment

LWisteria commented Jan 28, 2019

neko-suki commented Jan 28, 2019

neko-suki left a comment

jenkins-maekawa left a comment

jenkins-maekawa left a comment

jenkins-maekawa left a comment

jenkins-maekawa left a comment

neko-suki commented Jan 31, 2019

Support multiple GPUs #160

Support multiple GPUs #160

Conversation

LWisteria commented Jan 26, 2019 • edited

LWisteria commented Jan 26, 2019

jenkins-maekawa left a comment

Choose a reason for hiding this comment

jenkins-maekawa left a comment

Choose a reason for hiding this comment

jenkins-maekawa left a comment

Choose a reason for hiding this comment

jenkins-maekawa left a comment

Choose a reason for hiding this comment

LWisteria commented Jan 26, 2019 • edited

jenkins-maekawa left a comment

Choose a reason for hiding this comment

jenkins-maekawa left a comment

Choose a reason for hiding this comment

LWisteria commented Jan 28, 2019

neko-suki commented Jan 28, 2019

neko-suki left a comment

Choose a reason for hiding this comment

jenkins-maekawa left a comment

Choose a reason for hiding this comment

jenkins-maekawa left a comment

Choose a reason for hiding this comment

jenkins-maekawa left a comment

Choose a reason for hiding this comment

jenkins-maekawa left a comment

Choose a reason for hiding this comment

neko-suki commented Jan 31, 2019

LWisteria commented Jan 26, 2019 •

edited

LWisteria commented Jan 26, 2019 •

edited