Multiple plugin migrations and migrating empty repos when specified in the plan #87

dralley · 2020-02-04T18:48:02Z

closes: #5978
https://pulp.plan.io/issues/5978
closes: #5978
https://pulp.plan.io/issues/5978

dralley · 2020-02-04T18:50:58Z

Currently, I can only test this with pulpcore 3.0 and pulp_container 1.0, since the docker migration is broken with 3.1

ipanova · 2020-02-04T19:12:45Z

Daniel, have you tested with a MP where you migrate fully all iso plugin and a specific docker repo? Or specific iso repo and specific repo? I was thinking we'd need to change how we process the plan and get it to the structure where we split repos, importers, distributors per plugin.

…

On Tue, 4 Feb 2020, 19:48 Daniel Alley, ***@***.***> wrote: closes: #5978 https://pulp.plan.io/issues/5978 ------------------------------ You can view, comment on, or merge this pull request online at: #87 Commit Summary - Fix migration of multiple plugins simultaneously File Changes - *M* pulp_2to3_migration/app/migration.py <https://github.com/pulp/pulp-2to3-migration/pull/87/files#diff-0> (28) - *M* pulp_2to3_migration/app/serializers.py <https://github.com/pulp/pulp-2to3-migration/pull/87/files#diff-1> (2) Patch Links: - https://github.com/pulp/pulp-2to3-migration/pull/87.patch - https://github.com/pulp/pulp-2to3-migration/pull/87.diff — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#87?email_source=notifications&email_token=ABAOQ6HN5QKOEUFC75JBT23RBGZ6HA5CNFSM4KP4AFTKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IK7RKQA>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABAOQ6DQNOV3CLWNKVAVSCTRBGZ6HANCNFSM4KP4AFTA> .

dralley · 2020-02-04T19:32:02Z

I'm still working on testing it thoroughly (I wasn't really expecting it to be looked at today). Probably needs a WIP tag.

ipanova · 2020-02-04T19:38:15Z

No problem, I did not really look at it just quickly scrolled through.

…

On Tue, 4 Feb 2020, 20:32 Daniel Alley, ***@***.***> wrote: I'm still working on testing it thoroughly (I wasn't really expecting it to be looked at today). Probably needs a WIP tag. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#87?email_source=notifications&email_token=ABAOQ6EHDCMCIQWRLE2TG3LRBG7DHA5CNFSM4KP4AFTKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKY4TTA#issuecomment-582076876>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABAOQ6BW4RZWHVKH67FI6PLRBG7DHANCNFSM4KP4AFTA> .

dralley · 2020-02-04T20:03:07Z

@ipanova I used this plan to test:

http POST :24817/pulp/api/v3/migration-plans/ plan='{"plugins": [{"type": "iso", "repositories": [{"name": "filetest", "pulp2_importer_repository_id": "file", "repository_versi
ons": [{"pulp2_repository_id": "file", "pulp2_distributor_repository_ids": ["file"]}, {"pulp2_repository_id": "file2", "pulp2_distributor_repository_ids": ["file2"]}]}]}, {"type": "docker", "repositories": [{"name": "test-fixture-1", "rep
ository_versions": [{"pulp2_repository_id": "test-fixture-1", "pulp2_distributor_repository_ids": ["test-fixture-1"]}]}]}]}'

{
    "plan": {
        "plugins": [
            {
                "repositories": [
                    {
                        "name": "filetest",
                        "pulp2_importer_repository_id": "file",
                        "repository_versions": [
                            {
                                "pulp2_distributor_repository_ids": [
                                    "file"
                                ],
                                "pulp2_repository_id": "file"
                            },
                            {
                                "pulp2_distributor_repository_ids": [
                                    "file2"
                                ],
                                "pulp2_repository_id": "file2"
                            }
                        ]
                    }
                ],
                "type": "iso"
            },
            {
                "repositories": [
                    {
                        "name": "test-fixture-1",
                        "repository_versions": [
                            {
                                "pulp2_distributor_repository_ids": [
                                    "test-fixture-1"
                                ],
                                "pulp2_repository_id": "test-fixture-1"
                            }
                        ]
                    }
                ],
                "type": "docker"
            }
        ]
    },
    "pulp_created": "2020-02-04T19:50:07.646432Z",
    "pulp_href": "/pulp/api/v3/migration-plans/3dde9e85-3343-4605-a626-86ae56748c5e/"
}

And got the following results, which looks correct.

    "results": [
        {
            "is_migrated": true,
            "not_in_plan": false,
            "pulp2_object_id": "5e39b6c26e50c93273818358",
            "pulp2_repo_id": "file",
            "pulp3_distribution_hrefs": [
                "/pulp/api/v3/distributions/file/file/e25f9387-5d0d-4178-85ef-4dfb1ed062a5/"
            ],
            "pulp3_publication_href": "/pulp/api/v3/publications/file/file/91bd9fca-56d2-43ca-8bd5-eb221509331c/",
            "pulp3_remote_href": "/pulp/api/v3/remotes/file/file/c1c19710-fe43-40db-b65a-e0a1c4129a6f/",
            "pulp3_repository_href": "/pulp/api/v3/repositories/file/file/b2c84286-768f-4716-911d-69aba1bdba6a/",
            "pulp3_repository_version": "/pulp/api/v3/repositories/file/file/b2c84286-768f-4716-911d-69aba1bdba6a/versions/1/",
            "pulp_created": "2020-02-04T19:50:29.909310Z",
            "pulp_href": "/pulp/api/v3/pulp2repositories/fbfbbd3a-76ab-47da-a235-448b00320538/",
            "type": "iso"
        },
        {
            "is_migrated": true,
            "not_in_plan": false,
            "pulp2_object_id": "5e39b6c26e50c9327381835b",
            "pulp2_repo_id": "file2",
            "pulp3_distribution_hrefs": [
                "/pulp/api/v3/distributions/file/file/54a9858e-ddf3-4e9f-a5f3-fc06c1160057/"
            ],
            "pulp3_publication_href": "/pulp/api/v3/publications/file/file/1cd90201-e198-4e8c-8d48-94e4c4f1195a/",
            "pulp3_remote_href": "/pulp/api/v3/remotes/file/file/c1c19710-fe43-40db-b65a-e0a1c4129a6f/",
            "pulp3_repository_href": "/pulp/api/v3/repositories/file/file/b2c84286-768f-4716-911d-69aba1bdba6a/",
            "pulp3_repository_version": "/pulp/api/v3/repositories/file/file/b2c84286-768f-4716-911d-69aba1bdba6a/versions/2/",
            "pulp_created": "2020-02-04T19:50:29.896925Z",
            "pulp_href": "/pulp/api/v3/pulp2repositories/ddea27bd-abd5-42e2-abc6-b0bc5dd989c8/",
            "type": "iso"
        },
        {
            "is_migrated": true,
            "not_in_plan": false,
            "pulp2_object_id": "5e39b6c46e50c93275f65aee",
            "pulp2_repo_id": "test-fixture-1",
            "pulp3_distribution_hrefs": [
                "/pulp/api/v3/distributions/container/container/5d898e56-7396-4063-9869-c03e9b4cf4fd/"
            ],
            "pulp3_publication_href": null,
            "pulp3_remote_href": null,
            "pulp3_repository_href": "/pulp/api/v3/repositories/container/container/7599c647-6206-4d06-b923-9b63b54419e6/",
            "pulp3_repository_version": "/pulp/api/v3/repositories/container/container/7599c647-6206-4d06-b923-9b63b54419e6/versions/1/",
            "pulp_created": "2020-02-04T19:50:29.877117Z",
            "pulp_href": "/pulp/api/v3/pulp2repositories/80279b10-7a78-4d78-83d1-878437526229/",
            "type": "docker"
        }
    ]

dralley · 2020-02-04T20:06:13Z

@ipanova However, I think I've found a bug in the docker migrator. Using this plan (which only migrates docker, no iso) and a clean database, and pulpcore 3.0, pulp_container 1.0, and master branch of the migration plugin (ha! not making that mistake again...):

If you run a migration plan for the docker plugin twice (any plan), you get this

Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: Traceback (most recent call last):
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: return self.cursor.execute(sql, params)
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "container_blobmanifest_manifest_id_manifest_blo_05da3e8d_uniq"
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: DETAIL:  Key (manifest_id, manifest_blob_id)=(379a3152-5461-464d-a971-ef662026c04e, e17f96db-ab2b-41be-aa90-d719a72d4886) already exists.
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: The above exception was the direct cause of the following exception:
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: Traceback (most recent call last):
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/docker/migrator.py", line 80, in migrate_content_to_pulp3
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: await dm.create()
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/content.py", line 83, in create
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: await pipeline
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py", line 209, in create_pipeline
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: await asyncio.gather(*futures)
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py", line 43, in __call__
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: await self.run()
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/docker/migrator.py", line 178, in run
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: self.relate_blob(dc)
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/docker/migrator.py", line 214, in relate_blob
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: thru.save()
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/base.py", line 741, in save
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: force_update=force_update, update_fields=update_fields)
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/base.py", line 779, in save_base
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: force_update, using, update_fields,
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/base.py", line 870, in _save_table
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/base.py", line 908, in _do_insert
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: using=using, raw=raw)
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: return getattr(self.get_queryset(), name)(*args, **kwargs)
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/query.py", line 1186, in _insert
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: return query.get_compiler(using=using).execute_sql(return_id)
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1368, in execute_sql
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: cursor.execute(sql, params)
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: return executor(sql, params, many, context)
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: return self.cursor.execute(sql, params)
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: raise dj_exc_value.with_traceback(traceback) from exc_value
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: return self.cursor.execute(sql, params)
Feb 04 20:03:15 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[10867]: django.db.utils.IntegrityError: duplicate key value violates unique constraint "container_blobmanifest_manifest_id_manifest_blo_05da3e8d_uniq"

I've filed an issue for this: https://pulp.plan.io/issues/6099

dralley · 2020-02-04T22:35:54Z

Aside from that, a migration plan for all ISO and specific docker does seem to have issues though.

dralley · 2020-02-05T05:19:22Z

I got it mostly working with that example (blanket ISO migrate, specific docker), it still needs a little more work and testing.

dralley · 2020-02-05T05:38:22Z

The container plugin doesn't use publications, correct?

ipanova · 2020-02-05T09:28:56Z

The container plugin doesn't use publications, correct?

correct, it has only distributions

ipanova · 2020-02-05T10:31:04Z

@dralley #88 i submitted a PR to fix integrity error

dralley · 2020-02-05T22:03:35Z

Thanks @ipanova !

This should be working now. Tested with:

2 plugins, specific repos
2 plugins, docker specific, iso everything
2 plugins, docker everything, iso specific

dralley · 2020-02-07T02:26:41Z

Also, should now migrate empty repositories when they're specified.

ipanova · 2020-02-07T15:07:49Z

pulp_2to3_migration/app/migration.py

+                pulp3_distribution=None,
+                pulp3_publication=None,
+                not_in_plan=False,
+                pulp2_type_id=plugin.type


when i run the migration it does not create any publications or distributions, i think because of this line.
In [3]: Pulp2Distributor.objects.all()[0].pulp2_type_id
Out[3]: 'iso_distributor'

in plugin.type you will have 'iso'

you can fetch the types for distributors from the plugin.migrator.distriubutor_migrators.keys()

@dralley you applied this logic to pulp2repos and pulp2 distributors, what about pulp2importers?

Strange, I didn't seem to have that issue -- but I'll check again.

Strange, I didn't seem to have that issue -- but I'll check again.

https://paste.centos.org/view/0de7e0d2

ipanova · 2020-02-07T15:13:12Z

pulp_2to3_migration/app/serializers.py

@@ -156,6 +156,7 @@ class Pulp2RepositoriesSerializer(ModelSerializer):
    )
    pulp2_object_id = serializers.CharField(max_length=255)
    pulp2_repo_id = serializers.CharField()
+    type = serializers.CharField()


what about pulp2_repo_type to stay consistent with other fields names?

Should I rename the database field as well? It is just type

Would you also like the database field renamed? It is just type

dralley · 2020-02-07T18:55:01Z

@ipanova With continued testing I found a new bug :( I'll let you know when this is ready

dralley · 2020-02-07T19:56:04Z

Working again @ipanova

ipanova · 2020-02-07T22:51:35Z

pulp_2to3_migration/app/migration.py


-        for plugin in plan.get_plugin_plans():
            distributor_migrators.update(**plugin.migrator.distributor_migrators)


since we process distributors by plugin type now, i believe we can just assign directly distributor_migrators = plugin.migrator.distributor_migrators, no need to update the dict

ipanova · 2020-02-07T23:02:52Z

pulp_2to3_migration/app/migration.py

+                not_in_plan=False,
+                pulp2_type_id__in=distributor_types
+            )
+            pb.total = pulp2distributors_qs.count()


pb.total += pulp2distributors_qs.count()

ipanova · 2020-02-07T23:05:16Z

pulp_2to3_migration/app/migration.py

+                pulp2_type_id__in=distributor_types
+            )
+            pb.total = pulp2distributors_qs.count()
+            pb.save()


~~i think we should save the counters outside of the for loop~~ not that important

ipanova

I left a minor comment about counters. I have tested various corner cases and they do seems to work!
nice work 🌴 🍺

closes: #5978 https://pulp.plan.io/issues/5978

closes: #6070 https://pulp.plan.io/issues/6070

dralley force-pushed the multiple-plugin branch from 891cdc6 to 58e2133 Compare February 4, 2020 18:49

dralley added the WIP label Feb 4, 2020

dralley force-pushed the multiple-plugin branch from f4d293e to 1e01437 Compare February 5, 2020 21:59

dralley force-pushed the multiple-plugin branch 3 times, most recently from e7c1a23 to 62bf203 Compare February 5, 2020 22:48

dralley removed the WIP label Feb 6, 2020

dralley force-pushed the multiple-plugin branch 2 times, most recently from 6cb5ee9 to 523cb5c Compare February 7, 2020 02:22

dralley changed the title ~~Fix migration of multiple plugins simultaneously~~ Fix migration of multiple plugins simultaneously and migrate empty repos Feb 7, 2020

dralley changed the title ~~Fix migration of multiple plugins simultaneously and migrate empty repos~~ Multiple plugin migrations and migrating empty repos when specified in the plan Feb 7, 2020

ipanova reviewed Feb 7, 2020

View reviewed changes

dralley force-pushed the multiple-plugin branch 4 times, most recently from 3f7fd96 to baabd6c Compare February 7, 2020 16:17

dralley force-pushed the multiple-plugin branch 2 times, most recently from a3e07ea to 9317b43 Compare February 7, 2020 16:30

dralley force-pushed the multiple-plugin branch from 9317b43 to f797801 Compare February 7, 2020 19:55

ipanova reviewed Feb 7, 2020

View reviewed changes

ipanova approved these changes Feb 8, 2020

View reviewed changes

dralley added 2 commits February 8, 2020 00:15

Fix migration of multiple plugins simultaneously

b57a251

closes: #5978 https://pulp.plan.io/issues/5978

Migrate empty repos if specified in the migration plan

648bcfc

closes: #6070 https://pulp.plan.io/issues/6070

dralley force-pushed the multiple-plugin branch from f797801 to 648bcfc Compare February 8, 2020 05:16

dralley merged commit 84e1c01 into pulp:master Feb 8, 2020

dralley deleted the multiple-plugin branch February 8, 2020 05:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multiple plugin migrations and migrating empty repos when specified in the plan #87

Multiple plugin migrations and migrating empty repos when specified in the plan #87

dralley commented Feb 4, 2020 •

edited

dralley commented Feb 4, 2020

ipanova commented Feb 4, 2020 via email

dralley commented Feb 4, 2020

ipanova commented Feb 4, 2020 via email

dralley commented Feb 4, 2020 •

edited

dralley commented Feb 4, 2020 •

edited

dralley commented Feb 4, 2020 •

edited

dralley commented Feb 5, 2020 •

edited

dralley commented Feb 5, 2020

ipanova commented Feb 5, 2020

ipanova commented Feb 5, 2020

dralley commented Feb 5, 2020

dralley commented Feb 7, 2020

ipanova Feb 7, 2020

ipanova Feb 7, 2020

ipanova Feb 7, 2020

dralley Feb 7, 2020

ipanova Feb 7, 2020

ipanova Feb 7, 2020

dralley Feb 7, 2020

dralley Feb 7, 2020

ipanova Feb 7, 2020

dralley Feb 7, 2020

dralley commented Feb 7, 2020

dralley commented Feb 7, 2020

ipanova Feb 7, 2020

ipanova Feb 7, 2020

ipanova Feb 7, 2020 •

edited

ipanova left a comment


		for plugin in plan.get_plugin_plans():
		distributor_migrators.update(**plugin.migrator.distributor_migrators)

Multiple plugin migrations and migrating empty repos when specified in the plan #87

Multiple plugin migrations and migrating empty repos when specified in the plan #87

Conversation

dralley commented Feb 4, 2020 • edited

dralley commented Feb 4, 2020

ipanova commented Feb 4, 2020 via email

dralley commented Feb 4, 2020

ipanova commented Feb 4, 2020 via email

dralley commented Feb 4, 2020 • edited

dralley commented Feb 4, 2020 • edited

dralley commented Feb 4, 2020 • edited

dralley commented Feb 5, 2020 • edited

dralley commented Feb 5, 2020

ipanova commented Feb 5, 2020

ipanova commented Feb 5, 2020

dralley commented Feb 5, 2020

dralley commented Feb 7, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dralley commented Feb 7, 2020

dralley commented Feb 7, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ipanova Feb 7, 2020 • edited

Choose a reason for hiding this comment

ipanova left a comment

Choose a reason for hiding this comment

dralley commented Feb 4, 2020 •

edited

dralley commented Feb 4, 2020 •

edited

dralley commented Feb 4, 2020 •

edited

dralley commented Feb 4, 2020 •

edited

dralley commented Feb 5, 2020 •

edited

ipanova Feb 7, 2020 •

edited