Added twine upload support #367

gerrod3 · 2021-04-29T22:12:52Z

Lots to still work on, but twine support will finally be here.

Make endpoint available at /pulp_python/pypi/ as well
Replace constants with appropriate values from settings
Add twine-enable option to distribution (might need a better name for that)
Add tests
Add new workflows
Stretch - Add simple live api so you don't need publications.

fixes: #342

gerrod3 · 2021-04-29T22:32:16Z

pulp_python/app/pypi.py

+            session = metadata["session"]
+            start_time = datetime.now(tz=timezone.utc) + timedelta(seconds=5)
+            if not session.get("start"):
+                create_upload_task(session)
+            else:
+                sq = Session.objects.select_for_update(nowait=False).filter(pk=session.session_key)
+                with transaction.atomic():
+                    try:
+                        sq.first()
+                        if datetime.fromisoformat(session['start']) >= datetime.now(tz=timezone.utc):
+                            session['artifacts'].append((str(artifact.pk), file.name))
+                            session['start'] = str(start_time)
+                            session.modified = False
+                            session.save()
+                            log.info(f"Updated current task with new package {file.name}")
+                        else: raise DatabaseError
+                    except DatabaseError:
+                        session.cycle_key()
+                        create_upload_task(session)


Twine uploads each package in a separate post request, but they all share the same session. To group the uploaded packages into one task I pass the Session object to the task and have it wait 5 seconds until starting the task. If more packages are uploaded in those 5 seconds then the session is updated with those tasks and the start time of the task is pushed back. If a giant package is in the list of uploaded packages and the task starts before the package finishes uploading, then package's from that one and on will be added in a new task.

webknjaz · 2021-05-10T19:25:29Z

pulp_python/app/pypi.py

+        """
+        serializer = PyPIPackageUploadSerializer(data=request.data)
+        serializer.is_valid(raise_exception=True)
+        file, sha256 = serializer.validated_data["content"], serializer.validated_data["sha256_digest"]


FTR warehouse stores multiple hashes if you want to replicate that https://pypi.org/project/build/#copy-hash-modal-9cd0d632-b45a-435a-8d43-1b36ecdbd951

For every artifact in Pulp we store multiple hashes https://github.com/pulp/pulpcore/blob/master/pulpcore/app/models/content.py#L142, but sha256 is the main one we use. We disabled by default using md5 for security purposes and we don't calculate or store blake256 digests. I don't think it's necessary we check all 3 since sha256 is pretty secure and all the tools upload the package with the sha256.

pulp_python/app/pypi/serializers.py

webknjaz · 2021-06-01T22:46:20Z

pulp_python/app/pypi/serializers.py

+        try:
+            artifact.save()
+        except IntegrityError:
+            artifact = Artifact.objects.get(sha256=artifact.sha256)


This is inconsistent with Warehouse. The standard PyPI returns 403 on duplicate artifacts.

Yes, inconsistent with Warehouse, but not so for Pulp. One of Pulp's key features is artifact de-duplication. If an artifact is shared among multiple content or a piece of content is contained in multiple repositories, Pulp only stores the one artifact or content unit once. So for pulp python the content unit is the Python package and its artifact is the tarball or wheel itself. Since Pulp can have multiple repositories if the package is already present in Pulp, but not in the repository, then it should be treated as adding that package to that repository.

What if the artifact exists but the hash is different? It mustn't be possible to mutate the same dist. At least this case should error out.

The Artifact.init_and_validate will throw an error if the digest is wrong, so the same dist can't be mutated. However, if the package and its artifact already exist and you try to upload the same package with a different artifact, Pulp will just use the present artifact instead. This could be considered incorrect behavior.

you try to upload the same package with a different artifact

With the same hash or a different one?

Either or. Currently artifacts in Pulp are unique by their hash, but python packages in pulp_python are only unique by their filename. Since a goal of Pulp is to allow hosting multiple indexes, I think I'm going to need to change the global uniqueness constraint for python packages to be their hash, but still have the local uniqueness constraint for repositories be their filename.

I've changed the uniqueness constraint for packages to be their sha256. If you upload a package with the same hash as the one already in Pulp, but not in the repository you will simply add that package to that repository. If you try to upload a package with the same name as another in Pulp but not in the repository and with a different hash then a new package will be created and added to the repository. This means there can be two packages with the same name but different digests in Pulp, however they both can't be in the same repository. The sha256 of the package's metadata and its artifact are now linked and can't be mismatched.

This means there can be two packages with the same name but different digests in Pulp, however they both can't be in the same repository.

Awesome, this makes perfect sense 👍🏻 thx!

pulp_python/app/pypi/views.py

pulp_python/app/webserver_snippets/apache.conf

pulp_python/app/webserver_snippets/nginx.conf

docs/workflows/publish.rst

gerrod3

Some things I need to change and areas I would like feedback on.

pulp_python/app/pypi/serializers.py

pulp_python/app/pypi/views.py

gerrod3 · 2021-06-04T23:12:39Z

pulp_python/app/tasks/upload.py

-        repository = PythonRepository.objects.get(pk=repository_pk)
-        with repository.new_version() as new_version:
-            new_version.add_content(queryset)
+    @transaction.atomic()


This is already called from an atomic transaction. I wonder if this does anything.

It will create a new "savepoint" within the transaction https://docs.djangoproject.com/en/2.2/topics/db/transactions/#topics-db-transactions-savepoints

That functionality is actually a little controversial, SQLAlchemy and a few other ORMs keep different constructs for transactions and savepoints as the semantics are slightly different

docs/workflows/pypi.rst

dralley · 2021-06-11T03:18:37Z

docs/workflows/pypi.rst

+      "pulp_href": "/pulp/api/v3/distributions/python/pypi/e8438593-fd40-4654-8577-65398833c331/",
+      "pulp_created": "2021-06-03T20:04:18.233230Z",
+      "base_path": "my-pypi",
+      "base_url": "https://pulp3-source-fedora33.localhost.example.com/pypi/foo/",


dralley · 2021-06-11T03:21:42Z

pulp_python/app/models.py

@@ -43,6 +43,8 @@ class PythonDistribution(Distribution):

    TYPE = 'python'

+    allow_uploads = models.BooleanField(default=True)


It looks like the container registry plugin uses an entirely separate distribution type for this. I'm not sure what the reasons are, or if they would apply to the Python plugin as well. @ipanova could you weigh in?

Container registry uses same type of distribution, it however has 2 repo_types. Push repo type is auto-created during 'podman push', this is basically a read-only repo which you subsequently cannot use for mirroring.
I see that a python repo is being created in advance before python packages twine upload and you have just one repo type.
Is it possible to use same repo for mirroring and uploading? if there are no constraints in that area then i probably don't see the reason to apply this limitation, and if applying I would probably move this flag to the repo model instead.

That's my understanding of the way this PR works currently. Are there security reasons for the way the container plugin does it?

podman push operation creates a lot of intermediary repo-versions and only last repo-version is content compete meaning that in push-repo-type can be served only last version and it also means that rollback does not work.
Also if using one repo-type sync with 'mirror' mode would just remove everything what has been pushed previously by push api and this behavior can be undesirable. tldr; because of several subtleties it was easier and more explicit to have 2 repo types where one would be used for sync and another for push workflows. I think in registry world one would not use same repo for push and sync workflows and so that's why we have decided to bring the safeguards in a form of different repo types.

This however brings certain amount of complexity which we are figuring out how to minimize https://hackmd.io/z7i694VeTR-p4hgTk2Oocw#Food-for-thoughts

So it sounds like the container plugin has to deal with inter-content dependencies, and therefore intermediate repository versions are broken -- but because Python repositories only have one type of content, and one upload per content, that we therefore don't need to follow what the container plugin does with regards to separate repository / distribution types.

Thanks @ipanova, that is super helpful feedback.

pulp_python/app/pypi/views.py

dralley · 2021-06-11T03:36:52Z

pulp_python/app/tasks/upload.py

+        ContentArtifact.objects.create(
+            artifact=artifact, content=content, relative_path=filename
+        )
+        return content


This should probably be wrapped in a transaction, if it isn't already (and maybe it is)

dralley · 2021-06-11T03:39:43Z

pulp_python/tests/functional/api/test_pypi_apis.py

+        repo, distro = self._create_empty_repo_and_distribution()
+        url = urljoin(PYPI_HOST, distro.base_path + "/legacy/")
+        username, password = "admin", "password"
+        subprocess.run(


It's been a long, long time since I touched this functionality, but I would check to see if pulp_smash.cli.Client provides any extra functionality that makes it better suited than `subprocess1.

(This isn't a requirement, just a suggestion, I actually don't know if it does. We don't often drive a shell from functional tests)

Tried to use it originally, and it just wasn't well suited. The problem is that it runs the commands in the same container as the Pulp instance, but our CI installs the test_requirements in the VM running that container, so the pulp_smash CLI has no access to twine.

dralley

Fantastic work @gerrod3! This looks great. And the implementation seems very clean overall

I'll wait on Ina's feedback before approving, just in case there are good design reasons to go with multiple distribution types instead (like RBAC?)

fixes: pulp#342

gerrod3 commented Apr 29, 2021

View reviewed changes

gerrod3 force-pushed the twine branch 2 times, most recently from ee81c10 to a859785 Compare May 4, 2021 22:33

gerrod3 added this to the 3.3.0 milestone May 10, 2021

webknjaz reviewed May 10, 2021

View reviewed changes

gerrod3 force-pushed the twine branch from a859785 to e6bd547 Compare May 12, 2021 21:18

This was referenced May 13, 2021

Move simple writing code to utils #370

Merged

Add Python endpoints to Pulp API #373

Merged

gerrod3 modified the milestones: 3.3.0, 3.4.0 May 27, 2021

gerrod3 force-pushed the twine branch 5 times, most recently from 845f2e4 to 78bb9fb Compare June 1, 2021 22:07