As a user I can sync manifest lists.

closes #2384
https://pulp.plan.io/issues/2384

common/pulp_docker/common/constants.py

@@ -1,6 +1,9 @@
 BLOB_TYPE_ID = 'docker_blob'
 IMAGE_TYPE_ID = 'docker_image'
 MANIFEST_TYPE_ID = 'docker_manifest'
+MANIFEST_LIST_TYPE_ID = 'docker_manifest_list'
+MANIFEST_LIST_TYPE = 'list'
+MANIFEST_IMAGE_TYPE = 'image'
 TAG_TYPE_ID = 'docker_tag'
 IMPORTER_TYPE_ID = 'docker_importer'
 IMPORTER_CONFIG_FILE_NAME = 'server/plugins.conf.d/docker_importer.json'
@@ -51,6 +54,7 @@
 PUBLISH_STEP_BLOBS = 'publish_blobs'
 PUBLISH_STEP_IMAGES = 'publish_images'
 PUBLISH_STEP_MANIFESTS = 'publish_manifests'
+PUBLISH_STEP_MANIFEST_LISTS = 'publish_manifest_lists'
 PUBLISH_STEP_REDIRECT_FILE = 'publish_redirect_file'
 PUBLISH_STEP_TAGS = 'publish_tags'
 PUBLISH_STEP_OVER_HTTP = 'publish_images_over_http'

docs/tech-reference/distributor.rst

@@ -92,6 +92,44 @@ Supported keys
 
 .. _redirect_file:
 
+V4 Redirect File
+----------------
+
+For Docker v2 content, the distributors generate a json file with the details of the repository
+contents.
+
+The file is JSON formatted with the following keys
+
+* **type** *(string)* - the type of file. This will always be "pulp-docker-redirect".
+* **version** *(int)* - version of the format for the file. For Docker v2, that supports manifest
+                        list, this will be 3.
+* **repository** *(string)* - the name of the repository this file is describing.
+* **repo-registry-id** *(string)* - the name that will be used for this repository in the Docker
+  registry.
+* **url** *(string)* - the URL for accessing the repository content.
+* **schema2_data** *(array)* - an array of tags and digests that schema version 2 image manifests reference.
+* **manifest_list_data** *(array)* - an array of tags and digests that schema version 2 manifest lists reference.
+* **manifest_list_amd64_tags** *(object)* - dictionary of key-value pairs, where key is the tag of the manifest list,
+                                            and value is an array of the digest and schema version of the corresponding
+                                            image manifest for amd64 architecture and platform linux OS.
+* **protected** *(bool)* - whether or not the repository should be protected by an entitlement
+  certificate.
+
+Example Redirect File Contents::
+
+ {
+  "type":"pulp-docker-redirect",
+  "version":4,
+  "repository":"docker",
+  "repo-registry-id":"redhat/docker",
+  "url":"http://www.foo.com/docker",
+  "schema2_data":[],
+  "manifest_list_data":[],
+  "manifest_list_amd64_tags":{"latest": ["sha256:1234", 2]}
+  "protected": false
+ }
+
+
 V3 Redirect File
 ----------------
 

docs/tech-reference/tags.rst

@@ -6,60 +6,59 @@ v2
 
 Manifest Tags are stored as Pulp Units. Each Tag has a name, a manifest_digest
 (the digest of the Manifest that the Tag references), schema_version (the schema version
-for the manifest the Tag references) and a repo_id. A Tag's name, schema_version and repo_id
+for the manifest the Tag references), manifest_type( image manifest or manifest list the
+Tag references) and a repo_id. A Tag's name, schema_version, manifest_type and repo_id
 must be unique together so that in any given repository a Tag only references
-a single Manifest that uses either schema version 1 or schema version 2.
+a single Manifest(image or list).
 Here is an example of tag with name 'latest' within a repository::
 
     [
       {
-        "updated": "2017-02-09T15:52:46Z",
-        "repo_id": "synctest",
-        "created": "2017-02-09T15:52:46Z",
-        "_ns": "repo_content_units",
-        "unit_id": "19986269-4666-4dad-acbe-b0cce808bb21",
-        "unit_type_id": "docker_tag",
+        "updated": "2017-07-12T11:43:29Z", 
+        "repo_id": "man-list", 
+        "created": "2017-07-12T11:43:29Z", 
+        "unit_id": "98a4ba20-f60e-4b9d-8aa3-9bbe23030b4c", 
+        "unit_type_id": "docker_tag", 
         "_id": {
-          "$oid": "589c904e45ef487707c641fa"
-        },
-        "id": "589c904e45ef487707c641fa",
+          "$oid": "59660b6152e81521aead11c3"
+        }, 
         "metadata": {
-          "repo_id": "synctest",
-          "manifest_digest": "sha256:817a12c32a39bbe394944ba49de563e085f1d3c5266eb8e9723256bc4448680e",
-          "_ns": "units_docker_tag",
-          "_last_updated": 1486655449,
-          "schema_version": 2,
-          "pulp_user_metadata": {},
-          "_content_type_id": "docker_tag",
-          "_id": "19986269-4666-4dad-acbe-b0cce808bb21",
+          "repo_id": "man-list", 
+          "manifest_digest": "sha256:69fd2d3fa813bcbb3a572f1af80fe31a1710409e15dde91af79be62b37ab4f70", 
+          "manifest_type": "list", 
+          "_ns": "units_docker_tag", 
+          "_last_updated": 1499859809, 
+          "schema_version": 2, 
+          "pulp_user_metadata": {}, 
+          "_content_type_id": "docker_tag", 
+          "_id": "98a4ba20-f60e-4b9d-8aa3-9bbe23030b4c", 
           "name": "latest"
         }
-      },
+      }, 
       {
-        "updated": "2017-02-09T16:02:54Z",
-        "repo_id": "synctest",
-        "created": "2017-02-09T16:02:54Z",
-        "_ns": "repo_content_units",
-        "unit_id": "7f92741b-5379-4f13-8b43-0d3ebd9c5c25",
-        "unit_type_id": "docker_tag",
+        "updated": "2017-07-12T11:43:29Z", 
+        "repo_id": "man-list", 
+        "created": "2017-07-12T11:43:29Z", 
+        "unit_id": "cf56285e-1acd-4834-9dbd-35721b8964e6", 
+        "unit_type_id": "docker_tag", 
         "_id": {
-          "$oid": "589c92ae45ef487707c641fd"
-        },
-        "id": "589c92ae45ef487707c641fd",
+          "$oid": "59660b6152e81521aead11c2"
+        }, 
         "metadata": {
-          "repo_id": "synctest",
-          "manifest_digest": "sha256:c152ddeda2b828fbb610cb9e4cb121e1879dd5301d336f0a6c070b2844a0f56d",
-          "_ns": "units_docker_tag",
-          "_last_updated": 1486653137,
-          "schema_version": 1,
-          "pulp_user_metadata": {},
-          "_content_type_id": "docker_tag",
-          "_id": "7f92741b-5379-4f13-8b43-0d3ebd9c5c25",
+          "repo_id": "man-list", 
+          "manifest_digest": "sha256:7864a5b2d5ba865d0b3183071a4fc0dcaa365a599e90d5b54903076ed4ec5155", 
+          "manifest_type": "image", 
+          "_ns": "units_docker_tag", 
+          "_last_updated": 1499859809, 
+          "schema_version": 1, 
+          "pulp_user_metadata": {}, 
+          "_content_type_id": "docker_tag", 
+          "_id": "cf56285e-1acd-4834-9dbd-35721b8964e6", 
           "name": "latest"
         }
       }
     ]
-
+ 
 
 v1
 --

docs/user-guide/concepts.rst

@@ -7,13 +7,14 @@ Docker v2 Concepts
 Repository and Tags
 ^^^^^^^^^^^^^^^^^^^
 
-A Docker v2 repository is a collection of Blobs, Manifests, and Tags. Blobs are
-the layers that together make up a Docker image. The Manifest is the metadata
-that connects the Blobs together in the correct order, and it can also contain
-other metadata such as signatures. A Manifest can be tagged in a repository, and
-the Tag object is how this is accomplished in Pulp. So in short, a Tag
-references one Manifest (by digest) and a Manifest references N Blobs
-(also by digest).
+A Docker v2 repository is a collection of Blobs, Image Manifests, Manifest Lists
+and Tags. Blobs are the layers that together make up a Docker image. The Image
+Manifest is the metadata that connects the Blobs together in the correct order,
+and it can also contain other metadata such as signatures. A Manifest List is
+a list of Image manifests for one or more platforms. An Image Manifest or
+Manifest Listcan be tagged in a repository, and the Tag object is how this is
+accomplished in Pulp. So in short, a Tag references one Manifest(image or list)
+by digest same for a Image Manifest which references N Blobs (also by digest).
 
 .. note::
 
@@ -26,6 +27,7 @@ references one Manifest (by digest) and a Manifest references N Blobs
     what Tag name, users should rely on the ``name`` and ``manifest_digest``
     fields for Tag Units and not the Manifest ``tag`` field. In the Manifest v2
     schema, the ``tag`` field has been removed.
+    Since 3.0 fields ``tag`` and ``name`` and removed completely from Manifest model.
 
 Upload
 ^^^^^^

docs/user-guide/recipes.rst

@@ -373,62 +373,35 @@ to only one manifest.
 
 .. note::
 
-    Pulp now supports manifest schema 1 and schema 2 versions. So when tagging a manifest,
-    bear in mind that within a repo there could be two tags with the same name but pointing
-    to manifests with different schema versions.
+    Pulp now supports image manifest schema 1 and schema 2 versions, same as manifest lists schema 2.
+    So when tagging a manifest( image or list), bear in mind that within a repo there could be two
+    tags with the same name but pointing to manifests with different schema versions.
 
 
-For instance, suppose we have the following manifests::
+For instance, suppose we have the following image manifest that is tagged ::
 
-    $ pulp-admin docker repo search manifest --repo-id busybox
-    Created:      2016-11-10T16:27:30Z
-    Metadata:     
-      Digest:             sha256:4eccca494e527311eb4a4ebee1f90d9362971d882bb22fd7ded
-                          46d517129b1ac
-      Downloaded:         True
-      Fs Layers:          
-        Blob Sum: sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955
-                  b46d4
-        Blob Sum: sha256:191ff942861f5cfdc97ba2e76b5dec5f3894a9c21d6f88fbeaec2ea373c
-                  c657a
-        Blob Sum: sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955
-                  b46d4
-      Name:               library/busybox
-      Pulp User Metadata: 
-      Schema Version:     1
-      Tag:                latest
-    Repo Id:      busybox
-    Unit Id:      f064d1e9-0cbf-40ec-9648-85acbcc5e348
-    Unit Type Id: docker_manifest
-    Updated:      2016-11-10T16:27:30Z
+    pulp-admin docker repo search tag --repo-id man-list --str-eq='name=uclibc'
 
-    Created:      2016-11-10T16:27:30Z
+    Created:      2017-07-12T11:43:29Z
     Metadata:     
-      Digest:             sha256:c152ddeda2b828fbb610cb9e4cb121e1879dd5301d336f0a6c0
-                          70b2844a0f56d
-      Downloaded:         True
-      Fs Layers:          
-        Blob Sum: sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955
-                  b46d4
-        Blob Sum: sha256:8ddc19f16526912237dd8af81971d5e4dd0587907234be2b83e249518d5
-                  b673f
-      Name:               library/busybox
+      Manifest Digest:    sha256:26b0ddb0504097612cd7ed2265eade43f2490cd111a7cfcf7d1
+                          51dba83b20a5e
+      Manifest Type:      image
+      Name:               uclibc
       Pulp User Metadata: 
+      Repo Id:            man-list
       Schema Version:     1
-      Tag:                latest
-    Repo Id:      busybox
-    Unit Id:      f0663d57-a8d9-4093-a90c-5603280eafa3
-    Unit Type Id: docker_manifest
-    Updated:      2016-11-10T16:27:30Z
-
-If we have a tag named latest and it points to the first manifest with digest
-sha256:4ecca..., we can point it to the second manifest with the following
+    Repo Id:      man-list
+    Unit Id:      a37aa675-194c-4f07-925b-e1e12d98ad85
+    Unit Type Id: docker_tag
+    Updated:      2017-07-12T11:43:29Z
+
+If we have a tag named uclibc and it points to the manifest with digest
+sha256:26b0ddb0..., we can point it to the new manifest with the following
 command::
 
     $ pulp-admin docker repo tag --repo-id busybox --tag-name latest --manifest-digest sha256:c152ddeda2b828fbb610cb9e4cb121e1879dd5301d336f0a6c070b2844a0f56d
 
 We can also create a new tag and point it to the same manifest with::
 
     $ pulp-admin docker repo tag --repo-id busybox --tag-name 1.2 --manifest-digest sha256:c152ddeda2b828fbb610cb9e4cb121e1879dd5301d336f0a6c070b2844a0f56d
-
-

docs/user-guide/release-notes/3.0.x.rst

@@ -0,0 +1,19 @@
+3.0 Release Notes
+=================
+
+3.0.0
+-----
+
+Manifest List V2 schema version 2 is now supported. It can be normally synced into Pulp from
+docker registry, published and served by Crane. As a relust new unit type `ManifestList` was
+introduced in 3.0.
+Support for image manifest V2 schema version 1 and schema version 2  did not change.
+
+Publish directory structure for manifests has been changed to manage more effectively manifest
+lists. Now each manifest schema version has its own directory `manifests/1`, `manifests/2`
+and `manifests/list`. This change will not affect already published content, it will take place with
+the new publish action.
+
+A new `redirect file` format has been introduced to enable Crane to serve both schema versions.
+
+Existing command, docker repo tag now accepts instead of --manifest-digest --digest option.

docs/user-guide/release-notes/index.rst

@@ -6,6 +6,7 @@ Contents:
 .. toctree::
    :maxdepth: 2
 
+   3.0.x
    2.5.x
    2.4.x
    2.3.x