Add script to generate conda envs #367

bryevdv · 2022-09-20T21:18:09Z

This PR adds a script to generate a matrix of conda envs for

{osx, linux} x {py version} x {ctk version}

All of them are labeled "test" for their use, and reproduce the structure of the existing "test" env files.

This can certainly be made more sophisticated (e.g. handling openmpi, or splitting different envs for different use scenarios). Currently generates the following files:

~/build_test/legate.core bv/driver_refactor*
bldtest ❯ python scripts/generate-conda-envs.py 
------- environment-test-linux-py38-cuda-10.2.yaml
------- environment-test-linux-py38-cuda-11.0.yaml
------- environment-test-linux-py38-cuda-11.1.yaml
------- environment-test-linux-py38-cuda-11.2.yaml
------- environment-test-linux-py38-cuda-11.3.yaml
------- environment-test-linux-py38-cuda-11.4.yaml
------- environment-test-linux-py38-cuda-11.5.yaml
------- environment-test-linux-py38-cuda-11.6.yaml
------- environment-test-linux-py38-cuda-11.7.yaml
------- environment-test-linux-py38.yaml
------- environment-test-linux-py39-cuda-10.2.yaml
------- environment-test-linux-py39-cuda-11.0.yaml
------- environment-test-linux-py39-cuda-11.1.yaml
------- environment-test-linux-py39-cuda-11.2.yaml
------- environment-test-linux-py39-cuda-11.3.yaml
------- environment-test-linux-py39-cuda-11.4.yaml
------- environment-test-linux-py39-cuda-11.5.yaml
------- environment-test-linux-py39-cuda-11.6.yaml
------- environment-test-linux-py39-cuda-11.7.yaml
------- environment-test-linux-py39.yaml
------- environment-test-linux-py310-cuda-10.2.yaml
------- environment-test-linux-py310-cuda-11.0.yaml
------- environment-test-linux-py310-cuda-11.1.yaml
------- environment-test-linux-py310-cuda-11.2.yaml
------- environment-test-linux-py310-cuda-11.3.yaml
------- environment-test-linux-py310-cuda-11.4.yaml
------- environment-test-linux-py310-cuda-11.5.yaml
------- environment-test-linux-py310-cuda-11.6.yaml
------- environment-test-linux-py310-cuda-11.7.yaml
------- environment-test-linux-py310.yaml
------- environment-test-darwin-py38.yaml
------- environment-test-darwin-py39.yaml
------- environment-test-darwin-py310.yam

A sample env looks like:

name: legate-core-test
channels:
  - conda-forge
dependencies:

  - python=3.9

  # build
  - c-compiler
  - cmake>=3.24
  - cxx-compiler
  - gcc_linux-64 # [linux64]
  - git
  - make
  - ninja
  - openmpi
  - scikit-build>=0.14.1
  - setuptools>=61
  - sysroot_linux-64==2.17 # [linux64]
  - zlib

  # runtime
  - cffi
  - llvm-openmp
  - numpy>=1.22
  - opt_einsum
  - pyarrow>=5
  - scipy
  - typing_extensions

  # tests
  - clang-tools>=8
  - clang>=8
  - colorama
  - coverage
  - mock
  - mypy>=0.961
  - pre-commit
  - pytest-cov
  - pytest-lazy-fixture
  - pytest
  - types-docutils

  - pip
  - pip:

      # tests
      - tifffile

      # docs
      - jinja2
      - markdown<3.4.0
      - pydata-sphinx-theme
      - recommonmark
      - sphinx-copybutton
      - sphinx-markdown-tables
      - sphinx>=4.4.0

bryevdv · 2022-09-26T16:48:08Z

@manopapad @marcinz do you think it is desirable to add another dimension for {openmpi} ? If so is the only change to add/remove the openmpi package from the file? Or are there other packages that should be added/removed along with?

@marcinz also suggested {compilers} as a dimension. Do we want to pursue that at this point? If so, are toggling these two lines sufficient?

  - c-compiler
  - cxx-compile

Or are there additional packages? It's not clear e.g. how - gcc_linux-64 # [linux64] complements the lines above.

manopapad · 2022-09-29T00:17:58Z

Let's remove the following packages if the user doesn't want to use the conda compilers:

  - openmpi
  - c-compiler
  - cxx-compiler
  - gcc_linux-64
  - sysroot_linux-64

CC @m3vaz

Update: let's have 2 separate requirements, for openmpi (packages: openmpi) and compilers (packages: c-compiler, cxx-compiler). I think gcc_linux-64 and sysroot_linux64 should be removed altogether, see other comments.

scripts/generate-conda-envs.py

manopapad · 2022-09-29T01:32:26Z

@m3vaz should we try to protect developers from trying to use a more recent gcc than what the system nvcc will accept? I see the cudatoolkit package defines a minimum required libgcc-ng, but not a maximum. We could try to restrict the gcc version on our env files to a maximum, corresponding to the nvcc on the requested CTK version. This would assume the user will be asking for the same CTK as the system-wide installation supports.

bryevdv · 2022-09-29T21:58:58Z

@manopapad changes pushed, FYI this results in ~130 combinations

------- environment-test-linux-py38-cuda-10.2-compilers-with-openmpi.yaml
------- environment-test-linux-py38-cuda-10.2-compilers-without-openmpi.yaml
------- environment-test-linux-py38-cuda-10.2-no-compilers-with-openmpi.yaml
------- environment-test-linux-py38-cuda-10.2-no-compilers-without-openmpi.yaml
------- environment-test-linux-py38-cuda-11.0-compilers-with-openmpi.yaml
------- environment-test-linux-py38-cuda-11.0-compilers-without-openmpi.yaml
------- environment-test-linux-py38-cuda-11.0-no-compilers-with-openmpi.yaml
------- environment-test-linux-py38-cuda-11.0-no-compilers-without-openmpi.yaml
------- environment-test-linux-py38-cuda-11.1-compilers-with-openmpi.yaml
------- environment-test-linux-py38-cuda-11.1-compilers-without-openmpi.yaml
------- environment-test-linux-py38-cuda-11.1-no-compilers-with-openmpi.yaml
------- environment-test-linux-py38-cuda-11.1-no-compilers-without-openmpi.yaml
------- environment-test-linux-py38-cuda-11.2-compilers-with-openmpi.yaml
------- environment-test-linux-py38-cuda-11.2-compilers-without-openmpi.yaml
------- environment-test-linux-py38-cuda-11.2-no-compilers-with-openmpi.yaml
------- environment-test-linux-py38-cuda-11.2-no-compilers-without-openmpi.yaml
------- environment-test-linux-py38-cuda-11.3-compilers-with-openmpi.yaml
------- environment-test-linux-py38-cuda-11.3-compilers-without-openmpi.yaml
------- environment-test-linux-py38-cuda-11.3-no-compilers-with-openmpi.yaml
------- environment-test-linux-py38-cuda-11.3-no-compilers-without-openmpi.yaml
------- environment-test-linux-py38-cuda-11.4-compilers-with-openmpi.yaml
------- environment-test-linux-py38-cuda-11.4-compilers-without-openmpi.yaml
------- environment-test-linux-py38-cuda-11.4-no-compilers-with-openmpi.yaml
------- environment-test-linux-py38-cuda-11.4-no-compilers-without-openmpi.yaml
------- environment-test-linux-py38-cuda-11.5-compilers-with-openmpi.yaml
------- environment-test-linux-py38-cuda-11.5-compilers-without-openmpi.yaml
------- environment-test-linux-py38-cuda-11.5-no-compilers-with-openmpi.yaml
------- environment-test-linux-py38-cuda-11.5-no-compilers-without-openmpi.yaml
------- environment-test-linux-py38-cuda-11.6-compilers-with-openmpi.yaml
------- environment-test-linux-py38-cuda-11.6-compilers-without-openmpi.yaml
------- environment-test-linux-py38-cuda-11.6-no-compilers-with-openmpi.yaml
------- environment-test-linux-py38-cuda-11.6-no-compilers-without-openmpi.yaml
------- environment-test-linux-py38-cuda-11.7-compilers-with-openmpi.yaml
------- environment-test-linux-py38-cuda-11.7-compilers-without-openmpi.yaml
------- environment-test-linux-py38-cuda-11.7-no-compilers-with-openmpi.yaml
------- environment-test-linux-py38-cuda-11.7-no-compilers-without-openmpi.yaml
------- environment-test-linux-py38-compilers-with-openmpi.yaml
------- environment-test-linux-py38-compilers-without-openmpi.yaml
------- environment-test-linux-py38-no-compilers-with-openmpi.yaml
------- environment-test-linux-py38-no-compilers-without-openmpi.yaml
------- environment-test-linux-py39-cuda-10.2-compilers-with-openmpi.yaml
------- environment-test-linux-py39-cuda-10.2-compilers-without-openmpi.yaml
------- environment-test-linux-py39-cuda-10.2-no-compilers-with-openmpi.yaml
------- environment-test-linux-py39-cuda-10.2-no-compilers-without-openmpi.yaml
------- environment-test-linux-py39-cuda-11.0-compilers-with-openmpi.yaml
------- environment-test-linux-py39-cuda-11.0-compilers-without-openmpi.yaml
------- environment-test-linux-py39-cuda-11.0-no-compilers-with-openmpi.yaml
------- environment-test-linux-py39-cuda-11.0-no-compilers-without-openmpi.yaml
------- environment-test-linux-py39-cuda-11.1-compilers-with-openmpi.yaml
------- environment-test-linux-py39-cuda-11.1-compilers-without-openmpi.yaml
------- environment-test-linux-py39-cuda-11.1-no-compilers-with-openmpi.yaml
------- environment-test-linux-py39-cuda-11.1-no-compilers-without-openmpi.yaml
------- environment-test-linux-py39-cuda-11.2-compilers-with-openmpi.yaml
------- environment-test-linux-py39-cuda-11.2-compilers-without-openmpi.yaml
------- environment-test-linux-py39-cuda-11.2-no-compilers-with-openmpi.yaml
------- environment-test-linux-py39-cuda-11.2-no-compilers-without-openmpi.yaml
------- environment-test-linux-py39-cuda-11.3-compilers-with-openmpi.yaml
------- environment-test-linux-py39-cuda-11.3-compilers-without-openmpi.yaml
------- environment-test-linux-py39-cuda-11.3-no-compilers-with-openmpi.yaml
------- environment-test-linux-py39-cuda-11.3-no-compilers-without-openmpi.yaml
------- environment-test-linux-py39-cuda-11.4-compilers-with-openmpi.yaml
------- environment-test-linux-py39-cuda-11.4-compilers-without-openmpi.yaml
------- environment-test-linux-py39-cuda-11.4-no-compilers-with-openmpi.yaml
------- environment-test-linux-py39-cuda-11.4-no-compilers-without-openmpi.yaml
------- environment-test-linux-py39-cuda-11.5-compilers-with-openmpi.yaml
------- environment-test-linux-py39-cuda-11.5-compilers-without-openmpi.yaml
------- environment-test-linux-py39-cuda-11.5-no-compilers-with-openmpi.yaml
------- environment-test-linux-py39-cuda-11.5-no-compilers-without-openmpi.yaml
------- environment-test-linux-py39-cuda-11.6-compilers-with-openmpi.yaml
------- environment-test-linux-py39-cuda-11.6-compilers-without-openmpi.yaml
------- environment-test-linux-py39-cuda-11.6-no-compilers-with-openmpi.yaml
------- environment-test-linux-py39-cuda-11.6-no-compilers-without-openmpi.yaml
------- environment-test-linux-py39-cuda-11.7-compilers-with-openmpi.yaml
------- environment-test-linux-py39-cuda-11.7-compilers-without-openmpi.yaml
------- environment-test-linux-py39-cuda-11.7-no-compilers-with-openmpi.yaml
------- environment-test-linux-py39-cuda-11.7-no-compilers-without-openmpi.yaml
------- environment-test-linux-py39-compilers-with-openmpi.yaml
------- environment-test-linux-py39-compilers-without-openmpi.yaml
------- environment-test-linux-py39-no-compilers-with-openmpi.yaml
------- environment-test-linux-py39-no-compilers-without-openmpi.yaml
------- environment-test-linux-py310-cuda-10.2-compilers-with-openmpi.yaml
------- environment-test-linux-py310-cuda-10.2-compilers-without-openmpi.yaml
------- environment-test-linux-py310-cuda-10.2-no-compilers-with-openmpi.yaml
------- environment-test-linux-py310-cuda-10.2-no-compilers-without-openmpi.yaml
------- environment-test-linux-py310-cuda-11.0-compilers-with-openmpi.yaml
------- environment-test-linux-py310-cuda-11.0-compilers-without-openmpi.yaml
------- environment-test-linux-py310-cuda-11.0-no-compilers-with-openmpi.yaml
------- environment-test-linux-py310-cuda-11.0-no-compilers-without-openmpi.yaml
------- environment-test-linux-py310-cuda-11.1-compilers-with-openmpi.yaml
------- environment-test-linux-py310-cuda-11.1-compilers-without-openmpi.yaml
------- environment-test-linux-py310-cuda-11.1-no-compilers-with-openmpi.yaml
------- environment-test-linux-py310-cuda-11.1-no-compilers-without-openmpi.yaml
------- environment-test-linux-py310-cuda-11.2-compilers-with-openmpi.yaml
------- environment-test-linux-py310-cuda-11.2-compilers-without-openmpi.yaml
------- environment-test-linux-py310-cuda-11.2-no-compilers-with-openmpi.yaml
------- environment-test-linux-py310-cuda-11.2-no-compilers-without-openmpi.yaml
------- environment-test-linux-py310-cuda-11.3-compilers-with-openmpi.yaml
------- environment-test-linux-py310-cuda-11.3-compilers-without-openmpi.yaml
------- environment-test-linux-py310-cuda-11.3-no-compilers-with-openmpi.yaml
------- environment-test-linux-py310-cuda-11.3-no-compilers-without-openmpi.yaml
------- environment-test-linux-py310-cuda-11.4-compilers-with-openmpi.yaml
------- environment-test-linux-py310-cuda-11.4-compilers-without-openmpi.yaml
------- environment-test-linux-py310-cuda-11.4-no-compilers-with-openmpi.yaml
------- environment-test-linux-py310-cuda-11.4-no-compilers-without-openmpi.yaml
------- environment-test-linux-py310-cuda-11.5-compilers-with-openmpi.yaml
------- environment-test-linux-py310-cuda-11.5-compilers-without-openmpi.yaml
------- environment-test-linux-py310-cuda-11.5-no-compilers-with-openmpi.yaml
------- environment-test-linux-py310-cuda-11.5-no-compilers-without-openmpi.yaml
------- environment-test-linux-py310-cuda-11.6-compilers-with-openmpi.yaml
------- environment-test-linux-py310-cuda-11.6-compilers-without-openmpi.yaml
------- environment-test-linux-py310-cuda-11.6-no-compilers-with-openmpi.yaml
------- environment-test-linux-py310-cuda-11.6-no-compilers-without-openmpi.yaml
------- environment-test-linux-py310-cuda-11.7-compilers-with-openmpi.yaml
------- environment-test-linux-py310-cuda-11.7-compilers-without-openmpi.yaml
------- environment-test-linux-py310-cuda-11.7-no-compilers-with-openmpi.yaml
------- environment-test-linux-py310-cuda-11.7-no-compilers-without-openmpi.yaml
------- environment-test-linux-py310-compilers-with-openmpi.yaml
------- environment-test-linux-py310-compilers-without-openmpi.yaml
------- environment-test-linux-py310-no-compilers-with-openmpi.yaml
------- environment-test-linux-py310-no-compilers-without-openmpi.yaml
------- environment-test-darwin-py38-compilers-with-openmpi.yaml
------- environment-test-darwin-py38-compilers-without-openmpi.yaml
------- environment-test-darwin-py38-no-compilers-with-openmpi.yaml
------- environment-test-darwin-py38-no-compilers-without-openmpi.yaml
------- environment-test-darwin-py39-compilers-with-openmpi.yaml
------- environment-test-darwin-py39-compilers-without-openmpi.yaml
------- environment-test-darwin-py39-no-compilers-with-openmpi.yaml
------- environment-test-darwin-py39-no-compilers-without-openmpi.yaml
------- environment-test-darwin-py310-compilers-with-openmpi.yaml
------- environment-test-darwin-py310-compilers-without-openmpi.yaml
------- environment-test-darwin-py310-no-compilers-with-openmpi.yaml
------- environment-test-darwin-py310-no-compilers-without-openmpi.yaml

m3vaz · 2022-09-30T16:15:33Z

@m3vaz should we try to protect developers from trying to use a more recent gcc than what the system nvcc will accept
Could we set --alow-unsupported-compiler? We don't kick the tires on the newer compilers whenever they come out, but since our gcc compatibility typically lags a fair bit, it's common practice for users to just use the newer compilers anyway.

manopapad · 2022-09-30T16:23:41Z

Could we set --alow-unsupported-compiler? We don't kick the tires on the newer compilers whenever they come out, but since our gcc compatibility typically lags a fair bit, it's common practice for users to just use the newer compilers anyway.

That's a good suggestion, I'll mention it on the corresponding issue nv-legate/cunumeric#629, and we don't need to do anything about it here.

bryevdv · 2022-09-30T16:26:40Z

@manopapad should we split these into subdirectories? Perhaps by cuda version or os/cuda version?

manopapad · 2022-09-30T17:40:14Z

@manopapad should we split these into subdirectories? Perhaps by cuda version or os/cuda version?

I was thinking we can just distribute the script, and tell the user to generate a conda env by running it, with appropriate options for their system.

bryevdv · 2022-09-30T17:43:41Z

@manopapad 👍 I thought the intention was to check in identical sets in legate and downstream projects, but just distributing the script works too. But in that case we could add command like args to the script let users specify which config they want, in case they only want one or a subset.

manopapad · 2022-09-30T17:50:31Z

I thought the intention was to check in identical sets in legate and downstream projects

That's what I was also thinking initially, but if we're dealing with 100s of envs then I'm thinking it's probably less effort for the user to just invoke a script than to riffle through all the available options. Plus we're sparing our git history from that many files.

But in that case we could add command like args to the script let users specify which config they want, in case they only want one or a subset.

Yes, can we switch to a mode where the script just produces one env file, based on command line choices for the various dimensions?

* Perform consensus match more frequently for bigger free fields * Minor cleanup

bryevdv · 2022-09-30T21:14:23Z

@manopapad

bldtest ❯ python scripts/generate-conda-envs.py --help
usage: generate-conda-envs.py [-h] [--python {3.8,3.9,3.10}] [--ctk {none,10.2,11.0,11.1,11.2,11.3,11.4,11.5,11.6,11.7}] [--os {linux,osx}]
                              [--compilers | --no-compilers] [--openmpi | --no-openmpi]

optional arguments:
  -h, --help            show this help message and exit
  --python {3.8,3.9,3.10}
                        Python version to generate for, (default: all python versions)
  --ctk {none,10.2,11.0,11.1,11.2,11.3,11.4,11.5,11.6,11.7}
                        CTK version to generate for (default: all CTK versions)
  --os {linux,osx}      OS to generate for (default: all OSes)
  --compilers, --no-compilers
                        Whether to include conda compilers or not (default: both)
  --openmpi, --no-openmpi
                        Whether to include openmpi or not (default: both)

ex:

bldtest ❯ python scripts/generate-conda-envs.py --python 3.8 --no-compilers --openmpi --ctk "none" --os "linux"
--- generating: environment-test-linux-py38-no-compilers-with-openmpi.yaml

~/work/legate.core bv/env_gen_script*
bldtest ❯ python scripts/generate-conda-envs.py --no-compilers --openmpi --ctk "11.2"  
--- generating: environment-test-linux-py38-cuda-11.2-no-compilers-with-openmpi.yaml
--- generating: environment-test-linux-py39-cuda-11.2-no-compilers-with-openmpi.yaml
--- generating: environment-test-linux-py310-cuda-11.2-no-compilers-with-openmpi.yaml

scripts/generate-conda-envs.py

manopapad · 2022-10-24T21:46:02Z

@bryevdv I am testing the build on various machines using this new workflow. Once that's done we can merge this. In the meantime, can you confirm I didn't do anything stupid on 2ee39fd and c810722?

bryevdv · 2022-10-25T15:54:19Z

@manopapad Those look good, note that when we bump minpy to 3.9 we can get rid of the custom action

You may also specify an arbitrary action by passing an Action subclass or other object that implements the same interface. The BooleanOptionalAction is available in argparse and adds support for boolean actions such as --foo and --no-foo:

import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--foo', action=argparse.BooleanOptionalAction)
parser.parse_args(['--no-foo'])
Namespace(foo=False)

New in version 3.9.

Add script to generate conda envs

4b1383f

bryevdv added the category:task PR is a simple task and will not be included in release notes label Sep 20, 2022

bryevdv requested a review from manopapad September 20, 2022 21:18

manopapad added category:improvement PR introduces an improvement and will be classified as such in release notes and removed category:task PR is a simple task and will not be included in release notes labels Sep 22, 2022

manopapad reviewed Sep 29, 2022

View reviewed changes

scripts/generate-conda-envs.py Outdated Show resolved Hide resolved

manopapad reviewed Sep 29, 2022

View reviewed changes

scripts/generate-conda-envs.py Outdated Show resolved Hide resolved

manopapad reviewed Sep 29, 2022

View reviewed changes

scripts/generate-conda-envs.py Outdated Show resolved Hide resolved

bryevdv added 3 commits September 29, 2022 14:30

Merge branch 'branch-22.10' into bv/env_gen_script

a7b5b64

remove gcc and sysroot pkgs

0621ee7

split out openmpi and compilers options

4bed294

Adjust consensus match frequency based on field sizes (nv-legate#402)

10a015c

* Perform consensus match more frequently for bigger free fields * Minor cleanup

manopapad mentioned this pull request Sep 30, 2022

installation of OpenMPI from Conda results in incompatibilities with system MPI on many supercomputers nv-legate/cunumeric#630

Closed

add command line args for selection

e012e00

bryevdv and others added 2 commits September 30, 2022 14:15

Merge branch 'branch-22.10' into bv/env_gen_script

4dacaa0

help wording

2ba6120

manopapad changed the base branch from branch-22.10 to branch-22.12 September 30, 2022 21:59

manopapad added 2 commits October 10, 2022 13:41

Merge branch 'branch-22.12' into bv/env_gen_script

a59d694

Make script executable

adef1fd

manopapad added 6 commits October 10, 2022 13:52

Fixes for python 3.8

0a908f9

Remove old environment files

3a4943d

Unify file naming for "compilers" and "openmpi"

1436b69

Fix typo

f2fcc26

Remove optional ninja dependency

dea2c91

Not just for the core, include cunumeric also

6620575

manopapad reviewed Oct 10, 2022

View reviewed changes

scripts/generate-conda-envs.py Show resolved Hide resolved

manopapad reviewed Oct 10, 2022

View reviewed changes

scripts/generate-conda-envs.py Outdated Show resolved Hide resolved

manopapad and others added 14 commits October 10, 2022 22:37

Update build documentation

28a9375

Fix a file link

4cd486f

Fix formatting

1725e4c

Merge branch 'branch-22.12' into bv/env_gen_script

cf66c50

Merge branch 'branch-22.12' into bv/env_gen_script

2b4377d

remove typing_extensions dependency

e73f549

remove jinja dependency

c42bee8

slight vertical whitespace improvemetn

a6adafd

Merge branch 'branch-22.12' into bv/env_gen_script

f1551aa

Use custom BooleanFlag action, for the benefit of py3.8

2ee39fd

Update build instructions

59f763b

Fix intra-document reference

45cf325

Merge branch 'branch-22.12' into bv/env_gen_script

707f3dd

Revise file naming scheme

c810722

Update BUILD.md

3bd3b53

Merge branch 'branch-22.12' into bv/env_gen_script

a054c28

manopapad merged commit 5512ffb into nv-legate:branch-22.12 Oct 25, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add script to generate conda envs #367

Add script to generate conda envs #367

bryevdv commented Sep 20, 2022

bryevdv commented Sep 26, 2022 •

edited

manopapad commented Sep 29, 2022 •

edited

manopapad commented Sep 29, 2022

bryevdv commented Sep 29, 2022

m3vaz commented Sep 30, 2022

manopapad commented Sep 30, 2022

bryevdv commented Sep 30, 2022

manopapad commented Sep 30, 2022

bryevdv commented Sep 30, 2022 •

edited

manopapad commented Sep 30, 2022

bryevdv commented Sep 30, 2022 •

edited

manopapad commented Oct 24, 2022

bryevdv commented Oct 25, 2022

Add script to generate conda envs #367

Add script to generate conda envs #367

Conversation

bryevdv commented Sep 20, 2022

bryevdv commented Sep 26, 2022 • edited

manopapad commented Sep 29, 2022 • edited

manopapad commented Sep 29, 2022

bryevdv commented Sep 29, 2022

m3vaz commented Sep 30, 2022

manopapad commented Sep 30, 2022

bryevdv commented Sep 30, 2022

manopapad commented Sep 30, 2022

bryevdv commented Sep 30, 2022 • edited

manopapad commented Sep 30, 2022

bryevdv commented Sep 30, 2022 • edited

manopapad commented Oct 24, 2022

bryevdv commented Oct 25, 2022

bryevdv commented Sep 26, 2022 •

edited

manopapad commented Sep 29, 2022 •

edited

bryevdv commented Sep 30, 2022 •

edited

bryevdv commented Sep 30, 2022 •

edited