As a user, I can sync manifests schema version 2.

closes #2099

docs/tech-reference/distributor.rst

@@ -32,7 +32,7 @@ Supported keys
 ``protected``
  if "true" requests for this repo will be checked for an entitlement certificate authorizing
  the server url for this repository; if "false" no authorization checking will be done.
- This defaults to true.
+ This defaults to false.
 
 ``redirect-url``
  The server URL that will be used when generating the redirect map for connecting the Docker
@@ -92,6 +92,38 @@ Supported keys
 
 .. _redirect_file:
 
+V3 Redirect File
+----------------
+
+For Docker v2 content, the distributors generate a json file with the details of the repository
+contents.
+
+The file is JSON formatted with the following keys
+
+* **type** *(string)* - the type of file. This will always be "pulp-docker-redirect".
+* **version** *(int)* - version of the format for the file. For Docker v2, that supports manifest schema,
+                        this will be 3.
+* **repository** *(string)* - the name of the repository this file is describing.
+* **repo-registry-id** *(string)* - the name that will be used for this repository in the Docker
+  registry.
+* **url** *(string)* - the URL for accessing the repository content.
+* **schema2_data** *(array)* - an array of tags and digests that schema version 2 manifests reference.
+* **protected** *(bool)* - whether or not the repository should be protected by an entitlement
+  certificate.
+
+Example Redirect File Contents::
+
+ {
+  "type":"pulp-docker-redirect",
+  "version":3,
+  "repository":"docker",
+  "repo-registry-id":"redhat/docker",
+  "url":"http://www.foo.com/docker",
+  "schema2_data":[]}
+  "protected": false
+ }
+
+
 V2 Redirect File
 ----------------
 
@@ -100,12 +132,12 @@ contents.
 
 The file is JSON formatted with the following keys
 
-* **type** *(string)* - the type of the file. This will always be "pulp-docker-redirect".
+* **type** *(string)* - the type of file. This will always be "pulp-docker-redirect".
 * **version** *(int)* - version of the format for the file. For Docker v2, this will be 2.
 * **repository** *(string)* - the name of the repository this file is describing.
 * **repo-registry-id** *(string)* - the name that will be used for this repository in the Docker
   registry.
-* **url** *(string)* - the url for access to the repository's content.
+* **url** *(string)* - the URL for accessing the repository content.
 * **protected** *(bool)* - whether or not the repository should be protected by an entitlement
   certificate.
 
@@ -117,7 +149,7 @@ Example Redirect File Contents::
   "repository":"docker",
   "repo-registry-id":"redhat/docker",
   "url":"http://www.foo.com/docker",
-  "protected": true
+  "protected": false
  }
 
 
@@ -129,19 +161,19 @@ repository contents.
 
 The file is JSON formatted with the following keys
 
-* **type** *(string)* - the type of the file. This will always be "pulp-docker-redirect".
+* **type** *(string)* - the type of file. This will always be "pulp-docker-redirect".
 * **version** *(int)* - version of the format for the file. For Docker v1, this will be 1.
 * **repository** *(string)* - the name of the repository this file is describing.
 * **repo-registry-id** *(string)* - the name that will be used for this repository in the Docker
   registry.
-* **url** *(string)* - the url for access to the repository's content.
+* **url** *(string)* - the URL for accessing the repository content.
 * **protected** *(bool)* - whether or not the repository should be protected by an entitlement
   certificate.
 * **images** *(array)* - an array of objects describing each image/layer in the repository.
 
   * **id** *(str)* - the image id for the image.
 
-* **tags** *(obj)* - an object containing key, value paris of "tag-name":"image-id".
+* **tags** *(obj)* - an object containing key, value pairs of "tag-name":"image-id".
 
 Example Redirect File Contents::
 
@@ -151,7 +183,7 @@ Example Redirect File Contents::
   "repository":"docker",
   "repo-registry-id":"redhat/docker",
   "url":"http://www.foo.com/docker",
-  "protected": true,
+  "protected": false,
   "images":[
     {"id":"48e5f45168b97799ad0aafb7e2fef9fac57b5f16f6db7f67ba2000eb947637eb"},
     {"id":"511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158"},

docs/tech-reference/tags.rst

@@ -5,22 +5,60 @@ v2
 --
 
 Manifest Tags are stored as Pulp Units. Each Tag has a name, a manifest_digest
-(the digest of the Manifest that the Tag references), and a repo_id. A Tag's
-name and repo_id must be unique together so that in any given repository a Tag
-name only references a single Manifest. Here is an example tag from MongoDB::
-
-    {
-        "_id" : "4e50e89a-fbd9-454e-8f05-22a439698264",
-        "pulp_user_metadata" : {
+(the digest of the Manifest that the Tag references), schema_version (the schema version
+for the manifest the Tag references) and a repo_id. A Tag's name, schema_version and repo_id
+must be unique together so that in any given repository a Tag only references
+a single Manifest that uses either schema version 1 or schema version 2.
+Here is an example of tag with name 'latest' within a repository::
 
+    [
+      {
+        "updated": "2017-02-09T15:52:46Z",
+        "repo_id": "synctest",
+        "created": "2017-02-09T15:52:46Z",
+        "_ns": "repo_content_units",
+        "unit_id": "19986269-4666-4dad-acbe-b0cce808bb21",
+        "unit_type_id": "docker_tag",
+        "_id": {
+          "$oid": "589c904e45ef487707c641fa"
+        },
+        "id": "589c904e45ef487707c641fa",
+        "metadata": {
+          "repo_id": "synctest",
+          "manifest_digest": "sha256:817a12c32a39bbe394944ba49de563e085f1d3c5266eb8e9723256bc4448680e",
+          "_ns": "units_docker_tag",
+          "_last_updated": 1486655449,
+          "schema_version": 2,
+          "pulp_user_metadata": {},
+          "_content_type_id": "docker_tag",
+          "_id": "19986269-4666-4dad-acbe-b0cce808bb21",
+          "name": "latest"
+        }
+      },
+      {
+        "updated": "2017-02-09T16:02:54Z",
+        "repo_id": "synctest",
+        "created": "2017-02-09T16:02:54Z",
+        "_ns": "repo_content_units",
+        "unit_id": "7f92741b-5379-4f13-8b43-0d3ebd9c5c25",
+        "unit_type_id": "docker_tag",
+        "_id": {
+          "$oid": "589c92ae45ef487707c641fd"
         },
-        "_last_updated" : 1455043172,
-        "name" : "1-glibc",
-        "manifest_digest" : "sha256:d5ad6f092d781a71630261dc7ee6503bafb8c39e2c918e13c9e0765e10758a9b",
-        "repo_id" : "synctest",
-        "_ns" : "units_docker_tag",
-        "_content_type_id" : "docker_tag"
-    }
+        "id": "589c92ae45ef487707c641fd",
+        "metadata": {
+          "repo_id": "synctest",
+          "manifest_digest": "sha256:c152ddeda2b828fbb610cb9e4cb121e1879dd5301d336f0a6c070b2844a0f56d",
+          "_ns": "units_docker_tag",
+          "_last_updated": 1486653137,
+          "schema_version": 1,
+          "pulp_user_metadata": {},
+          "_content_type_id": "docker_tag",
+          "_id": "7f92741b-5379-4f13-8b43-0d3ebd9c5c25",
+          "name": "latest"
+        }
+      }
+    ]
 
 
 v1

docs/user-guide/recipes.rst

@@ -363,6 +363,13 @@ the tag we specify does not exist, it will be created. If the tag exists
 however, it will be updated as tag name is unique per repository and can point
 to only one manifest.
 
+.. note::
+
+    Pulp now supports manifest schema 1 and schema 2 versions. So when tagging a manifest,
+    bear in mind that within a repo there could be two tags with the same name but pointing
+    to manifests with different schema versions.
+
+
 For instance, suppose we have the following manifests::
 
     $ pulp-admin docker repo search manifest --repo-id busybox

docs/user-guide/release-notes/2.4.x.rst

@@ -0,0 +1,16 @@
+2.4 Release Notes
+=================
+
+2.4.0
+-----
+
+Manifest V2 schema version 2 is now supported. It can be normally synced into Pulp from
+docker registry, published and served by Crane.
+Support for manifest V2 schema version 1 did not change.
+
+Publish directory structure for manifests has been changed to manage more effectively manifest
+schema versions. Now each manifest schema version has its own directory `manifests/1` and
+`manifests/2`. This change will not affect already published content, it will take place with
+the new publish action.
+
+A new `redirect file` format has been introduced to enable Crane to serve both schema versions.

docs/user-guide/release-notes/index.rst

@@ -6,6 +6,7 @@ Contents:
 .. toctree::
    :maxdepth: 2
 
+   2.4.x
    2.3.x
    2.2.x
    2.1.x

extensions_admin/pulp_docker/extensions/admin/upload.py

@@ -118,6 +118,7 @@ def generate_unit_key(self, filename, **kwargs):
 
         tag_name = kwargs[TAG_NAME_OPTION.keyword]
         repo_id = kwargs[std_options.OPTION_REPO_ID.keyword]
+
         return {'name': tag_name, 'repo_id': repo_id}
 
     def generate_metadata(self, filename, **kwargs):

plugins/etc/httpd/conf.d/pulp_docker.conf

@@ -15,6 +15,12 @@ Alias /pulp/docker/v2 /var/www/pub/docker/v2/web
     SSLRequireSSL
     Options FollowSymlinks Indexes
 </Directory>
+<Directory /var/www/pub/docker/v2/web/*/manifests/2>
+    Header set Docker-Distribution-API-Version "registry/2.0"
+    Header set Content-Type "application/vnd.docker.distribution.manifest.v2+json"
+    SSLRequireSSL
+    Options FollowSymlinks Indexes
+</Directory>
 
 # Docker v1
 Alias /pulp/docker/v1 /var/www/pub/docker/v1/web