Create Serializers and Viewsets for Content #291

asmacdo · 2018-12-19T23:03:29Z

This commit leaves POST (create) unimplemented as well as Filters which
are each complex enough that they warrent a separate issue and PR.

https://pulp.plan.io/issues/3989
fixes #3989

asmacdo · 2018-12-19T23:04:39Z

Added WIP so I remember to add all the help_texts

pep8speaks · 2018-12-20T14:33:04Z

Hello @asmacdo! Thanks for updating the PR.

Cheers ! There are no PEP8 issues in this Pull Request. 🍻

Comment last updated on January 07, 2019 at 15:39 Hours UTC

pulp_docker/app/serializers.py

ipanova

@asmacdo what about size property on the serializers?
i think the plan was to retrieve this information from the Artifact/RemoteArtifact

pulp_docker/app/serializers.py

asmacdo · 2018-12-21T14:59:50Z

pulp_docker/app/models.py

@@ -123,6 +123,9 @@ class BlobManifestBlob(models.Model):
    manifest_blob = models.ForeignKey(
        ManifestBlob, related_name='manifest_blobs', on_delete=models.CASCADE)

+    class Meta:
+        unique_together = ('manifest', 'manifest_blob')


After adding the viewsets, I found that the many-to-many relationships were being written multiple times. This made the serialization really ugly, and I wasn't sure it was working correctly, so I went ahead and fixed this problem since it was simple.

If preferred, I can break these changes into a different PR, but making the change makes it easier to check that the viewsets/serializers were working as expected

asmacdo · 2018-12-21T15:11:27Z

pulp_docker/app/viewsets.py

+        """
+        Create a new ManifestListTag from a request.
+        """
+        raise NotImplementedError()


Since the parent ViewSet includes POST (create), it is necessary to override this-- and because the upload functionality is sufficiently complex, I think it should be done in a separate issue/PR. I'll file that issue before I merge.

dkliban · 2018-12-21T18:29:21Z

pulp_docker/app/serializers.py

+    Even though Docker content can only have 1 Artifact, this field should be used with
+    `many=True` because the Artifact is related with the ContentArtifact through table, and when
+    the same Artifact is served at multiple relative_paths, there are multiple associations of
+    between a given Content and Artifact.


When does an artifact get served at multiple relative paths?

The only one that I investigated was tag names

This commit leaves POST (create) unimplemented as well as Filters which are each complex enough that they warrent a separate issue and PR. https://pulp.plan.io/issues/3989 fixes #3989

asmacdo · 2019-01-07T15:29:02Z

Here's how a blob serializers, FYI

    {
        "_artifact": {
            "_href": "/pulp/api/v3/artifacts/331/",
            "size": 715296
        },
        "_created": "2019-01-07T13:11:25.349266Z",
        "_href": "/pulp/api/v3/content/docker/blobs/300/",
        "_type": "pulp_docker.manifest-blob",
        "digest": "sha256:0ffadd58f2a61468f527cc4f0fc45272ee4a1a428abe014546c89de2aa6a0eb5",
        "media_type": "application/vnd.docker.image.rootfs.diff.tar.gzip"
    }

Previously, ImageManifests that were tagged were had multiple relations to the same artifact, 1 for the sha and 1 for each tag. This was caused by sharing the DeclarativeArtifact between the tags and the ImageManifest during sync. After those relations were pruned, update the serializers and models to a single-artifact Content. [noissue]

dkliban

I tested this out and it worked for me. Thanks!

asmacdo · 2019-01-08T14:45:24Z

There was some concern about some Manifests that associate the same blob more than once. I am thinking this problem was environmental however. I tested this with busybox and got the expected results.

If anyone encounters this problem, please have a look at the db using shell_plus.

 multiple_blobs = ImageManifest.objects.annotate(a_count=Count('blobs')).filter(a_count__gt=1)

This query returned 1 result, which was a Manifest that indeed had 2 blobs, but they were distinct.


~/d/devel ❯ http --auth admin:admin http://pulp3:8000/pulp/api/v3/content/docker/manifests/5/
HTTP/1.1 200 OK
Allow: GET, HEAD, OPTIONS
Content-Length: 503
Content-Type: application/json
Date: Tue, 08 Jan 2019 14:44:14 GMT
Server: WSGIServer/0.2 CPython/3.7.2
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "_artifact": {
        "_href": "/pulp/api/v3/artifacts/10/",
        "size": 711
    },
    "_created": "2019-01-08T14:31:11.893252Z",
    "_href": "/pulp/api/v3/content/docker/manifests/5/",
    "_type": "pulp_docker.manifest",
    "blobs": [
        "/pulp/api/v3/content/docker/blobs/106/",
        "/pulp/api/v3/content/docker/blobs/111/"
    ],
    "config_blob": "/pulp/api/v3/content/docker/blobs/109/",
    "digest": "sha256:19d091b7368bf6538f40439f9530b878ce3831f8ff46c8bd95d296509d1303e5",
    "media_type": "application/vnd.docker.distribution.manifest.v2+json",
    "schema_version": 2
}

~/d/devel ❯ http --auth admin:admin http://pulp3:8000/pulp/api/v3/content/docker/blobs/106/
HTTP/1.1 200 OK
Allow: GET, HEAD, OPTIONS
Content-Length: 337
Content-Type: application/json
Date: Tue, 08 Jan 2019 14:44:23 GMT
Server: WSGIServer/0.2 CPython/3.7.2
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "_artifact": {
        "_href": "/pulp/api/v3/artifacts/140/",
        "size": 32
    },
    "_created": "2019-01-08T14:31:15.884055Z",
    "_href": "/pulp/api/v3/content/docker/blobs/106/",
    "_type": "pulp_docker.manifest-blob",
    "digest": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4",
    "media_type": "application/vnd.docker.image.rootfs.diff.tar.gzip"
}

~/d/devel ❯ http --auth admin:admin http://pulp3:8000/pulp/api/v3/content/docker/blobs/111/
HTTP/1.1 200 OK
Allow: GET, HEAD, OPTIONS
Content-Length: 341
Content-Type: application/json
Date: Tue, 08 Jan 2019 14:44:27 GMT
Server: WSGIServer/0.2 CPython/3.7.2
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "_artifact": {
        "_href": "/pulp/api/v3/artifacts/144/",
        "size": 686275
    },
    "_created": "2019-01-08T14:31:16.009351Z",
    "_href": "/pulp/api/v3/content/docker/blobs/111/",
    "_type": "pulp_docker.manifest-blob",
    "digest": "sha256:f5c4e6c495776bf839ba15ab1a68dc7ef2f491ec4c258431a107445812ecd65d",
    "media_type": "application/vnd.docker.image.rootfs.diff.tar.gzip"
}

asmacdo · 2019-01-08T16:54:12Z

FWIW, I also tested this by running multiple syncs, and got the same results as above.

ipanova · 2019-01-08T17:04:16Z

@asmacdo i confirm that it was env issue

asmacdo added the Work In Progress label Dec 19, 2018

asmacdo force-pushed the add-content-viewsets-serializers branch from 5c2e903 to 017c5c7 Compare December 20, 2018 14:33

asmacdo force-pushed the add-content-viewsets-serializers branch 3 times, most recently from f89748d to 7f12da3 Compare December 20, 2018 15:27

asmacdo removed the Work In Progress label Dec 20, 2018

asmacdo commented Dec 20, 2018

View reviewed changes

pulp_docker/app/serializers.py Outdated Show resolved Hide resolved

ipanova reviewed Dec 20, 2018

View reviewed changes

pulp_docker/app/serializers.py Outdated Show resolved Hide resolved

asmacdo force-pushed the add-content-viewsets-serializers branch from 7f12da3 to b2d5736 Compare December 21, 2018 14:40

asmacdo commented Dec 21, 2018

View reviewed changes

asmacdo force-pushed the add-content-viewsets-serializers branch from b2d5736 to cdc3237 Compare December 21, 2018 15:20

dkliban reviewed Dec 21, 2018

View reviewed changes

Create Serializers and Viewsets for Content

dff93a5

This commit leaves POST (create) unimplemented as well as Filters which are each complex enough that they warrent a separate issue and PR. https://pulp.plan.io/issues/3989 fixes #3989

asmacdo force-pushed the add-content-viewsets-serializers branch 2 times, most recently from 1610d92 to 2bf5485 Compare January 7, 2019 14:37

asmacdo force-pushed the add-content-viewsets-serializers branch from 2bf5485 to 5544e14 Compare January 7, 2019 15:39

dkliban approved these changes Jan 8, 2019

View reviewed changes

asmacdo merged commit 4f97829 into pulp:master Jan 8, 2019

Create Serializers and Viewsets for Content #291

Create Serializers and Viewsets for Content #291

asmacdo commented Dec 19, 2018

asmacdo commented Dec 19, 2018

pep8speaks commented Dec 20, 2018 •

edited

ipanova left a comment

asmacdo Dec 21, 2018

asmacdo Dec 21, 2018

dkliban Dec 21, 2018

asmacdo Dec 21, 2018

asmacdo commented Jan 7, 2019

dkliban left a comment

asmacdo commented Jan 8, 2019

asmacdo commented Jan 8, 2019

ipanova commented Jan 8, 2019

Create Serializers and Viewsets for Content #291

Create Serializers and Viewsets for Content #291

Conversation

asmacdo commented Dec 19, 2018

asmacdo commented Dec 19, 2018

pep8speaks commented Dec 20, 2018 • edited

Comment last updated on January 07, 2019 at 15:39 Hours UTC

ipanova left a comment

Choose a reason for hiding this comment

asmacdo Dec 21, 2018

Choose a reason for hiding this comment

asmacdo Dec 21, 2018

Choose a reason for hiding this comment

dkliban Dec 21, 2018

Choose a reason for hiding this comment

asmacdo Dec 21, 2018

Choose a reason for hiding this comment

asmacdo commented Jan 7, 2019

dkliban left a comment

Choose a reason for hiding this comment

asmacdo commented Jan 8, 2019

asmacdo commented Jan 8, 2019

ipanova commented Jan 8, 2019

pep8speaks commented Dec 20, 2018 •

edited