'import perspective' complains about undefined symbol on Linux (Manjaro) #1120

pgagarinov · 2020-07-11T16:42:44Z

Bug Report

`❯ ipython
Python 3.7.7 (default, May 7 2020, 21:25:33)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import perspective
CRITICAL:root:Failed to import C++ bindings for Perspective probably as it could not be built for your architecture (check install logs for more details).
You can still use PerspectiveWidget in client mode using JupyterLab.
Traceback (most recent call last):
File "/home/peter/.conda/envs/redash/lib/python3.7/site-packages/perspective/libpsp.py", line 21, in
from .table import * # noqa: F401, F403
File "/home/peter/.conda/envs/redash/lib/python3.7/site-packages/perspective/table/init.py", line 9, in
from .table import Table
File "/home/peter/.conda/envs/redash/lib/python3.7/site-packages/perspective/table/table.py", line 11, in
from .view import View
File "/home/peter/.conda/envs/redash/lib/python3.7/site-packages/perspective/table/view.py", line 15, in
from ._data_formatter import to_format, _parse_format_options
File "/home/peter/.conda/envs/redash/lib/python3.7/site-packages/perspective/table/_data_formatter.py", line 12, in
from .libbinding import get_data_slice_zero, get_data_slice_one, get_data_slice_two,
ImportError: /home/peter/.conda/envs/redash/lib/python3.7/site-packages/perspective/table/libpsp.so: undefined symbol: _ZNK5arrow6Buffer8ToStringB5cxx11Ev
`

Log from pip install -v perspective-python

build.log

Steps to Reproduce:

install miniconda //to /opt/miniconda
rm -rf ./.cache
rm -rf ./.conda
conda create -n redash python=3.7 -y
conda activate redash
pip install requests jupyter-server notebook ipython pyarrow==0.16.0
rm ./build.log
pip install -v perspective-python --log build.log
ipython
import perspective

Expected Result:

I expect no errors

Actual Result:

In [1]: import perspective CRITICAL:root:Failed to import C++ bindings for Perspective probably as it could not be built for your architecture (check install logs for more details). You can still use PerspectiveWidgetin client mode using JupyterLab. Traceback (most recent call last): File "/home/peter/.conda/envs/redash/lib/python3.7/site-packages/perspective/libpsp.py", line 21, in <module> from .table import * # noqa: F401, F403 File "/home/peter/.conda/envs/redash/lib/python3.7/site-packages/perspective/table/__init__.py", line 9, in <module> from .table import Table File "/home/peter/.conda/envs/redash/lib/python3.7/site-packages/perspective/table/table.py", line 11, in <module> from .view import View File "/home/peter/.conda/envs/redash/lib/python3.7/site-packages/perspective/table/view.py", line 15, in <module> from ._data_formatter import to_format, _parse_format_options File "/home/peter/.conda/envs/redash/lib/python3.7/site-packages/perspective/table/_data_formatter.py", line 12, in <module> from .libbinding import get_data_slice_zero, get_data_slice_one, get_data_slice_two, \ ImportError: /home/peter/.conda/envs/redash/lib/python3.7/site-packages/perspective/table/libpsp.so: undefined symbol: _ZNK5arrow6Buffer8ToStringB5cxx11Ev

Environment:

❯ neofetch
██████████████████ ████████ peter@darkstar
██████████████████ ████████ --------------
██████████████████ ████████ OS: Manjaro Linux x86_64
██████████████████ ████████ Kernel: 5.4.44-1-MANJARO
████████ ████████ Uptime: 9 days, 8 hours, 29 mins
████████ ████████ ████████ Packages: 1674 (pacman), 8 (flatpak)
████████ ████████ ████████ Shell: zsh 5.8
████████ ████████ ████████ Resolution: 3840x1600
████████ ████████ ████████ DE: Plasma
████████ ████████ ████████ WM: KWin
████████ ████████ ████████ WM Theme: Breeze
████████ ████████ ████████ Theme: Breath2 [Plasma], Breath [GTK2/3]
████████ ████████ ████████ Icons: breath2 [Plasma], breath2 [GTK2/3]
████████ ████████ ████████ Terminal: terminator
CPU: Intel i7-8750H (12) @ 4.100GHz
GPU: Intel UHD Graphics 630
Memory: 9915MiB / 32020MiB

Miniconda 4.8.3

cmake [Installed] 3.17.3-1 extra
A cross-platform open-source make system

Log from pip install -v perspective-python

build.log

The text was updated successfully, but these errors were encountered:

Yuri2b · 2020-07-11T19:45:41Z

Try pip install --no-clean -v perspective-python to keep built files and copying them from the build folder into your /home/peter/.conda/envs/redash/lib/python3.7/site-packages/perspective/table/
The same issue on Windows: #1108

pgagarinov · 2020-07-11T22:32:09Z

@Yuri2b
Thanks for the advice but it doesn't work ((
Doesn't work because libbinding.so, libpsp.so and libtbb.so are already in place and except for these 3 there are no other build files to copy to /home/peter/.conda/envs/redash/lib/python3.7/site-packages/perspective/table/. In #1108 copying dll's seems to help but on Linux there are no extra dlls to copy while so files are all already in place.

sc1f · 2020-07-11T22:38:11Z

@pgagarinov what is the output of ldd libpsp.so and ldd libbinding.so? This error looks like Perspective is linked to a version of pyarrow built on Python 2, although I'm not sure how that could have happened.

Additionally, would you be able to post the logs from pip install -v perspective-python? Paging @timkpaine as well

pgagarinov · 2020-07-11T22:39:30Z

By the way, I've deciphered the undefined symbol:

❯ c++filt _ZNK5arrow6Buffer8ToStringB5cxx11Ev
arrow::Buffer::ToStringabi:cxx11 const

So looks like perspective cannot find arrow::Buffer::ToString function... Any ideas?

pgagarinov · 2020-07-11T22:41:37Z

❯ ldd libpsp.so
linux-vdso.so.1 (0x00007ffe061a2000)
libarrow_python.so.16 => /home/peter/.conda/envs/redash/lib/python3.7/site-packages/perspective/table/./../../pyarrow/libarrow_python.so.16 (0x00007fc42b982000)
libarrow.so.16 => /home/peter/.conda/envs/redash/lib/python3.7/site-packages/perspective/table/./../../pyarrow/libarrow.so.16 (0x00007fc42a765000)
libpython3.8.so.1.0 => /usr/lib/libpython3.8.so.1.0 (0x00007fc42a38f000)
libtbb.so.2 => /usr/lib/libtbb.so.2 (0x00007fc42a349000)
libstdc++.so.6 => /home/peter/.conda/envs/redash/lib/libstdc++.so.6 (0x00007fc42a1d5000)
libm.so.6 => /usr/lib/libm.so.6 (0x00007fc42a08e000)
libgcc_s.so.1 => /home/peter/.conda/envs/redash/lib/libgcc_s.so.1 (0x00007fc42a07a000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007fc429eb3000)
/usr/lib64/ld-linux-x86-64.so.2 (0x00007fc42c08b000)
libdl.so.2 => /usr/lib/libdl.so.2 (0x00007fc429ead000)
librt.so.1 => /usr/lib/librt.so.1 (0x00007fc429ea2000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007fc429e80000)
libutil.so.1 => /usr/lib/libutil.so.1 (0x00007fc429e79000)
❯ ldd libbinding.so
linux-vdso.so.1 (0x00007ffda9b4f000)
libarrow_python.so.16 => /home/peter/.conda/envs/redash/lib/python3.7/site-packages/perspective/table/./../../pyarrow/libarrow_python.so.16 (0x00007f1306f59000)
libarrow.so.16 => /home/peter/.conda/envs/redash/lib/python3.7/site-packages/perspective/table/./../../pyarrow/libarrow.so.16 (0x00007f1305d3c000)
libpython3.8.so.1.0 => /usr/lib/libpython3.8.so.1.0 (0x00007f1305966000)
libpsp.so => /home/peter/.conda/envs/redash/lib/python3.7/site-packages/perspective/table/./libpsp.so (0x00007f1305611000)
libtbb.so.2 => /usr/lib/libtbb.so.2 (0x00007f13055cb000)
libstdc++.so.6 => /home/peter/.conda/envs/redash/lib/libstdc++.so.6 (0x00007f1305455000)
libm.so.6 => /usr/lib/libm.so.6 (0x00007f1305310000)
libgcc_s.so.1 => /home/peter/.conda/envs/redash/lib/libgcc_s.so.1 (0x00007f13052fc000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007f1305135000)
libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f130512f000)
librt.so.1 => /usr/lib/librt.so.1 (0x00007f1305124000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f1305100000)
/usr/lib64/ld-linux-x86-64.so.2 (0x00007f13073e8000)
libutil.so.1 => /usr/lib/libutil.so.1 (0x00007f13050fb000)

pgagarinov · 2020-07-11T22:43:24Z

@pgagarinov what is the output of ldd libpsp.so and ldd libbinding.so? This error looks like Perspective is linked to a version of pyarrow built on Python 2, although I'm not sure how that could have happened.

Additionally, would you be able to post the logs from pip install -v perspective-python? Paging @timkpaine as well

The log is attached in the very beginning of this bug report (build.log file).

pgagarinov · 2020-07-12T07:11:22Z

@sc1f
Looks like perspective is ended up being linked to python3.8

❯ ldd libpsp.so
linux-vdso.so.1 (0x00007ffe061a2000)
libarrow_python.so.16 => /home/peter/.conda/envs/redash/lib/python3.7/site-packages/perspective/table/./../../pyarrow/libarrow_python.so.16 (0x00007fc42b982000)
libarrow.so.16 => /home/peter/.conda/envs/redash/lib/python3.7/site-packages/perspective/table/./../../pyarrow/libarrow.so.16 (0x00007fc42a765000)
libpython3.8.so.1.0 => /usr/lib/libpython3.8.so.1.0 (0x00007fc42a38f000)

which is a system-wide python in Arch (and Manjaro) right now:
❯ sudo pacman -Q python
python 3.8.3-1

while in fact the python version installed within the conda environment is 3.7.7:
❯ conda list python
Name Version Build Channel
ipython 7.16.1 pypi_0 pypi
ipython-genutils 0.2.0 pypi_0 pypi
perspective-python 0.5.1 pypi_0 pypi
python 3.7.7 hcff3b4d_5
python-dateutil 2.8.1 pypi_0 pypi

pgagarinov · 2020-07-12T09:11:27Z

@timkpaine
@sc1f

I've identified the root of the problem - I did not have "cmake" installed within the conda environment and because of that the system-wide cmake was used. The system-wide cmake used system-wide path to libraries including libpython3.8.so.1.0

After installing cmake in conda environment I have the issue similar to #1108:

  Building wheel for perspective-python (PEP 517) ... error
  ERROR: Failed building wheel for perspective-python
Failed to build perspective-python
ERROR: Could not build wheels for perspective-python which use PEP 517 and cannot be installed directly
Exception information:
Traceback (most recent call last):
  File "/home/peter/.conda/envs/redash/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 188, in _main
    status = self.run(options, args)
  File "/home/peter/.conda/envs/redash/lib/python3.7/site-packages/pip/_internal/cli/req_command.py", line 185, in wrapper
    return func(self, options, args)
  File "/home/peter/.conda/envs/redash/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 378, in run
    ", ".join(r.name for r in pep517_build_failures)))
pip._internal.exceptions.InstallationError: Could not build wheels for perspective-python which use PEP 517 and cannot be installed directly
Removed build tracker: '/tmp/pip-req-tracker-d0_bnufh'

Build log

build6.log

conda list

``` >conda list _libgcc_mutex 0.1 main attrs 19.3.0 pypi_0 pypi backcall 0.2.0 pypi_0 pypi bleach 3.1.5 pypi_0 pypi bzip2 1.0.8 h516909a_2 conda-forge ca-certificates 2020.6.20 hecda079_0 conda-forge certifi 2020.6.20 py37hc8dfbb8_0 conda-forge chardet 3.0.4 pypi_0 pypi cmake 3.14.0 h52cb24c_0 decorator 4.4.2 pypi_0 pypi defusedxml 0.6.0 pypi_0 pypi entrypoints 0.3 pypi_0 pypi expat 2.2.9 he1b5a44_2 conda-forge idna 2.10 pypi_0 pypi importlib-metadata 1.7.0 pypi_0 pypi ipykernel 5.3.2 pypi_0 pypi ipython 7.16.1 pypi_0 pypi ipython-genutils 0.2.0 pypi_0 pypi jedi 0.17.1 pypi_0 pypi jinja2 2.11.2 pypi_0 pypi jsonschema 3.2.0 pypi_0 pypi jupyter-client 6.1.5 pypi_0 pypi jupyter-core 4.6.3 pypi_0 pypi jupyter-server 0.3.0 pypi_0 pypi krb5 1.17.1 hfafb76e_1 conda-forge ld_impl_linux-64 2.33.1 h53a641e_7 libcurl 7.71.1 hcdd3856_1 conda-forge libedit 3.1.20191231 h7b6447c_0 libffi 3.3 he6710b0_2 libgcc-ng 9.1.0 hdf63c60_0 libssh2 1.9.0 hab1572f_3 conda-forge libstdcxx-ng 9.1.0 hdf63c60_0 markupsafe 1.1.1 pypi_0 pypi mistune 0.8.4 pypi_0 pypi nbconvert 5.6.1 pypi_0 pypi nbformat 5.0.7 pypi_0 pypi ncurses 6.2 he6710b0_1 notebook 6.0.3 pypi_0 pypi numpy 1.19.0 pypi_0 pypi openssl 1.1.1g h516909a_0 conda-forge packaging 20.4 pypi_0 pypi pandocfilters 1.4.2 pypi_0 pypi parso 0.7.0 pypi_0 pypi pexpect 4.8.0 pypi_0 pypi pickleshare 0.7.5 pypi_0 pypi pip 20.1.1 py37_1 prometheus-client 0.8.0 pypi_0 pypi prompt-toolkit 3.0.5 pypi_0 pypi ptyprocess 0.6.0 pypi_0 pypi pyarrow 0.16.0 pypi_0 pypi pygments 2.6.1 pypi_0 pypi pyparsing 2.4.7 pypi_0 pypi pyrsistent 0.16.0 pypi_0 pypi python 3.7.7 hcff3b4d_5 python-dateutil 2.8.1 pypi_0 pypi python_abi 3.7 1_cp37m conda-forge pyzmq 19.0.1 pypi_0 pypi readline 8.0 h7b6447c_0 requests 2.24.0 pypi_0 pypi rhash 1.3.8 h1ba5d50_0 send2trash 1.5.0 pypi_0 pypi setuptools 47.3.1 py37_0 six 1.15.0 pypi_0 pypi sqlite 3.32.3 h62c20be_0 terminado 0.8.3 pypi_0 pypi testpath 0.4.4 pypi_0 pypi tk 8.6.10 hbc83047_0 tornado 6.0.4 pypi_0 pypi traitlets 4.3.3 pypi_0 pypi urllib3 1.25.9 pypi_0 pypi wcwidth 0.2.5 pypi_0 pypi webencodings 0.5.1 pypi_0 pypi wheel 0.34.2 py_1 conda-forge xz 5.2.5 h7b6447c_0 zipp 3.1.0 pypi_0 pypi zlib 1.2.11 h7b6447c_3 ```

conda info

❯ conda info

     active environment : redash
    active env location : /home/peter/.conda/envs/redash
            shell level : 2
       user config file : /home/peter/.condarc
 populated config files : 
          conda version : 4.8.3
    conda-build version : not installed
         python version : 3.7.6.final.0
       virtual packages : __glibc=2.31
       base environment : /opt/miniconda  (read only)
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /opt/miniconda/pkgs
                          /home/peter/.conda/pkgs
       envs directories : /home/peter/.conda/envs
                          /opt/miniconda/envs
               platform : linux-64
             user-agent : conda/4.8.3 requests/2.24.0 CPython/3.7.6 Linux/5.4.44-1-MANJARO manjaro/20.0.3 glibc/2.31
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False

Attempt to compile on Manjaro with python3.8:

I also ran a separate experiment by trying to build perspective outside of conda directly inside OS, no luck, same error.

Build log

build8.log

sc1f · 2020-07-12T12:41:54Z

Which version of CMake do you have installed through conda? It’s saying that policy CMP0094 is not recognized by this version of CMake - it’s in 3.15 onwards. We use CMP0094 to differentiate between python executable versions when searching for python in CMake.

sc1f · 2020-07-13T16:16:22Z

In the direct build (build8.log) it looks like it can't find PyArrow:

2020-07-12T12:11:08,709   CMake Error at /usr/share/cmake-3.17/Modules/FindPackageHandleStandardArgs.cmake:164 (message):
2020-07-12T12:11:08,709     Could NOT find PyArrow (missing: PYTHON_PYARROW_INCLUDE_DIR
2020-07-12T12:11:08,709     PYTHON_PYARROW_LIBRARY_DIR) (found version "")

Looks like it's using /usr/lib to find the site-packages folder - but PyArrow is in .local/lib.

pgagarinov · 2020-07-13T18:41:55Z

@sc1f

Which version of CMake do you have installed through conda? It’s saying that policy CMP0094 is not recognized by this version of CMake - it’s in 3.15 onwards. We use CMP0094 to differentiate between python executable versions when searching for python in CMake.

It was cmake 3.14. I upgraded to 3.17 and was able to compile perspective. However this time I have the same error

ImportError: /home/peter/.conda/envs/redash/lib/python3.7/site-packages/perspective/table/libpsp.so: undefined symbol: _ZNK5arrow6Buffer8ToStringB5cxx11Ev

ldd shows

❯ ldd /home/peter/.conda/envs/redash/lib/python3.7/site-packages/perspective/table/libpsp.so
	linux-vdso.so.1 (0x00007f24d4d8b000)
	libarrow_python.so.16 => /home/peter/.conda/envs/redash/lib/python3.7/site-packages/perspective/table/../../pyarrow/libarrow_python.so.16 (0x00007f24d467f000)
	libarrow.so.16 => /home/peter/.conda/envs/redash/lib/python3.7/site-packages/perspective/table/../../pyarrow/libarrow.so.16 (0x00007f24d3462000)
	libpython3.8.so.1.0 => /usr/lib/libpython3.8.so.1.0 (0x00007f24d308c000)
	libtbb.so.2 => /usr/lib/libtbb.so.2 (0x00007f24d3046000)
	libstdc++.so.6 => /home/peter/.conda/envs/redash/lib/libstdc++.so.6 (0x00007f24d2ed2000)
	libm.so.6 => /usr/lib/libm.so.6 (0x00007f24d2d8b000)
	libgcc_s.so.1 => /home/peter/.conda/envs/redash/lib/libgcc_s.so.1 (0x00007f24d2d77000)
	libc.so.6 => /usr/lib/libc.so.6 (0x00007f24d2bb0000)
	/usr/lib64/ld-linux-x86-64.so.2 (0x00007f24d4d8c000)
	libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f24d2baa000)
	librt.so.1 => /usr/lib/librt.so.1 (0x00007f24d2b9f000)
	libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f24d2b7d000)
	libutil.so.1 => /usr/lib/libutil.so.1 (0x00007f24d2b76000)

Strangely enough this time perspective was linked to system-wide python3.8 even though I installed cmake inside conda
environment.
I have also tried to install cmake via pip instead of conda (inside the same conda environment), this leads to the following error

  Traceback (most recent call last):
    File "/home/peter/.conda/envs/redash/bin/cmake", line 5, in <module>
      from cmake import cmake
  ModuleNotFoundError: No module named 'cmake'
  Traceback (most recent call last):

As for the direct build - I've double checked that pyarrow is installed via system-wide pip:

❯ pip show pyarrow
Name: pyarrow
Version: 0.16.0
Summary: Python library for Apache Arrow
Home-page: https://arrow.apache.org/
Author: None
Author-email: None
License: Apache License, Version 2.0
Location: /home/peter/.local/lib/python3.8/site-packages
Requires: numpy, six
Required-by:

So looks like the build doesn't see pyarrow even though it is installed...

I could have tried to install python-arrow system package via pacman but unfortunately it is of version 0.15.1:

❯ pamac search python-arrow
python-arrow                                                                                                                                                                                                                             0.15.5-1  community 
    Better dates and times for Python

As a temporary solution I ended up using perspective 0.5.1 package from conda-forge channel, it works without any problems. The only thing that concerns me is that conda-forge::perspective 0.5.1 is linked to arrow-cpp 0.15.1 instead of arrow-cpp 0.16. I've created a ticket for that: conda-forge/perspective-feedstock#5

pgagarinov · 2020-07-17T12:57:21Z

@timkpaine
I see that you marked this as a question, not be a bug. I tend to believe this is a bug because Arch is popular among software developers and there should be at least a single way to compile perspective on Arch Linux but I haven't found any so far.

I suspect that https://perspective.finos.org/docs/md/development.html#ubuntudebian assumes that system-wide python is 3.7(as in Debian Buster), not 3.8. Thus https://perspective.finos.org/docs/md/development.html should either state the build cannot work on Linux with Python 3.8 OR should provide the compilation instructions. Same for conda - if the build script is not compatible with conda environment https://perspective.finos.org/docs/md/development.html should state that clearly.

timkpaine · 2020-07-17T13:37:17Z

@pgagarinov we support building on windows/linux/Mac python3.6, 3.7, and 3.8 but we can't support the infinite user configurations possible. We also don't support conda outside of what conda-forge provides right now. So "question" is more appropriate here since this platform or configuration is not in the subset we support directly.

timkpaine · 2020-09-11T21:04:06Z

We've made some improvements in this space in #1193 and #1163

timkpaine added the question Questions about use, potential features, or improvements label Jul 15, 2020

timkpaine mentioned this issue Aug 23, 2020

fix for virtualenvs/envs with multiple py3 installed, allow for up to arrow 0.17 #1163

Merged

pgagarinov mentioned this issue Aug 26, 2020

CMake picking up different paths for Python executable, libs, includes, not respecting version var #1162

Closed

timkpaine closed this as completed Sep 11, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

'import perspective' complains about undefined symbol on Linux (Manjaro) #1120

'import perspective' complains about undefined symbol on Linux (Manjaro) #1120

pgagarinov commented Jul 11, 2020 •

edited

Yuri2b commented Jul 11, 2020

pgagarinov commented Jul 11, 2020 •

edited

sc1f commented Jul 11, 2020

pgagarinov commented Jul 11, 2020

pgagarinov commented Jul 11, 2020 •

edited

pgagarinov commented Jul 11, 2020 •

edited

pgagarinov commented Jul 12, 2020 •

edited

pgagarinov commented Jul 12, 2020 •

edited

sc1f commented Jul 12, 2020

sc1f commented Jul 13, 2020 •

edited

pgagarinov commented Jul 13, 2020 •

edited

pgagarinov commented Jul 17, 2020 •

edited

timkpaine commented Jul 17, 2020

timkpaine commented Sep 11, 2020

'import perspective' complains about undefined symbol on Linux (Manjaro) #1120

'import perspective' complains about undefined symbol on Linux (Manjaro) #1120

Comments

pgagarinov commented Jul 11, 2020 • edited

Bug Report

Log from pip install -v perspective-python

Steps to Reproduce:

Expected Result:

Actual Result:

Environment:

Log from pip install -v perspective-python

Yuri2b commented Jul 11, 2020

pgagarinov commented Jul 11, 2020 • edited

sc1f commented Jul 11, 2020

pgagarinov commented Jul 11, 2020

pgagarinov commented Jul 11, 2020 • edited

pgagarinov commented Jul 11, 2020 • edited

pgagarinov commented Jul 12, 2020 • edited

pgagarinov commented Jul 12, 2020 • edited

Build log

conda list

conda info

Attempt to compile on Manjaro with python3.8:

Build log

sc1f commented Jul 12, 2020

sc1f commented Jul 13, 2020 • edited

pgagarinov commented Jul 13, 2020 • edited

pgagarinov commented Jul 17, 2020 • edited

timkpaine commented Jul 17, 2020

timkpaine commented Sep 11, 2020

pgagarinov commented Jul 11, 2020 •

edited

pgagarinov commented Jul 11, 2020 •

edited

pgagarinov commented Jul 11, 2020 •

edited

pgagarinov commented Jul 11, 2020 •

edited

pgagarinov commented Jul 12, 2020 •

edited

pgagarinov commented Jul 12, 2020 •

edited

sc1f commented Jul 13, 2020 •

edited

pgagarinov commented Jul 13, 2020 •

edited

pgagarinov commented Jul 17, 2020 •

edited