manifest has messed up encoding

[TomasTomecek]

I have pushed busybox image to my locally running distribution. When I curl a manifest for this image, the encoding is scrambled:

000013b0: 635c 222c 5c22 2328 6e6f 7029 204d 4149  c\",\"#(nop) MAI
000013c0: 4e54 4149 4e45 5220 4ac3 a972 c3b4 6d65  NTAINER J..r..me
000013d0: 2050 6574 617a 7a6f 6e69 205c 5c75 3030   Petazzoni \\u00
000013e0: 3363 6a65 726f 6d65 4064 6f63 6b65 722e  3cjerome@docker.
000013f0: 636f 6d5c 5c75 3030 3365 5c22 5d2c 5c22  com\\u003e\"],\"

n/sh\",\"-c\",\"#(nop) MAINTAINER Jérôme Petazzoni \\u003cjerome@docker.com\\u003e\"],\"Image\":\"51

As you can see, some characters are encoded and some are still unicode.

Registry is running from docker hub:

REPOSITORY           TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
docker.io/registry   2                   2f1ef7702586        7 weeks ago         220.6 MB

Docker is from my distribution:

$ docker version                                                                                                                                     
Client:
 Version:      1.8.1.fc22
 API version:  1.20
 Package Version: docker-1.8.1-3.git32b8b25.fc22.x86_64
 Go version:   go1.4.2
 Git commit:   32b8b25/1.8.1
 Built:        
 OS/Arch:      linux/amd64

Server:
 Version:      1.8.1.fc22
 API version:  1.20
 Package Version: 
 Go version:   go1.4.2
 Git commit:   32b8b25/1.8.1
 Built:        
 OS/Arch:      linux/amd64

[RichardScothern]

Which manifest is this? (I was unable to find it on the hub)

Is the encoding causing an issue for you?

[TomasTomecek]

busybox

Yes, it is. It affects digest of manifest.

[RichardScothern]

The reason I ask is that my busybox manifest looks different.

docker pull busybox
docker tag busybox localhost:5000/busybox
docker push localhost:5000/busybox

$ curl localhost:5000/v2/busybox/manifests/latest

{
   "schemaVersion": 1,
   "name": "busybox",
   "tag": "latest",
   "architecture": "amd64",
   "fsLayers": [
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:9d7588d3c0635b53bd9a7dcd40bdf5d2d32cd3fb919c3a29ec2febbc2449eb19"
      }
   ],
   "history": [
      {
         "v1Compatibility": "{\"id\":\"d7057cb020844f245031d27b76cb18af05db1cc3a96a29fa7777af75f5ac91a3\",\"parent\":\"cfa753dfea5e68a24366dfba16e6edf573daa447abf65bc11619c1a98a3aff54\",\"created\":\"2015-09-21T20:15:47.866196515Z\",\"container\":\"7f652467f9e6d1b3bf51172868b9b0c2fa1c711b112f4e987029b1624dd6295f\",\"container_config\":{\"Hostname\":\"5f8e0e129ff1\",\"Domainname\":\"\",\"User\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"ExposedPorts\":null,\"PublishService\":\"\",\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":null,\"Cmd\":[\"/bin/sh\",\"-c\",\"#(nop) CMD [\\\"sh\\\"]\"],\"Image\":\"cfa753dfea5e68a24366dfba16e6edf573daa447abf65bc11619c1a98a3aff54\",\"Volumes\":null,\"VolumeDriver\":\"\",\"WorkingDir\":\"\",\"Entrypoint\":null,\"NetworkDisabled\":false,\"MacAddress\":\"\",\"OnBuild\":null,\"Labels\":null},\"docker_version\":\"1.8.2\",\"config\":{\"Hostname\":\"5f8e0e129ff1\",\"Domainname\":\"\",\"User\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"ExposedPorts\":null,\"PublishService\":\"\",\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":null,\"Cmd\":[\"sh\"],\"Image\":\"cfa753dfea5e68a24366dfba16e6edf573daa447abf65bc11619c1a98a3aff54\",\"Volumes\":null,\"VolumeDriver\":\"\",\"WorkingDir\":\"\",\"Entrypoint\":null,\"NetworkDisabled\":false,\"MacAddress\":\"\",\"OnBuild\":null,\"Labels\":null},\"architecture\":\"amd64\",\"os\":\"linux\",\"Size\":0}\n"
      },
      {
         "v1Compatibility": "{\"id\":\"cfa753dfea5e68a24366dfba16e6edf573daa447abf65bc11619c1a98a3aff54\",\"created\":\"2015-09-21T20:15:47.433616227Z\",\"container\":\"5f8e0e129ff1e03bbb50a8b6ba7636fa5503c695125b1c392490d8aa113e8cf6\",\"container_config\":{\"Hostname\":\"5f8e0e129ff1\",\"Domainname\":\"\",\"User\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"ExposedPorts\":null,\"PublishService\":\"\",\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":null,\"Cmd\":[\"/bin/sh\",\"-c\",\"#(nop) ADD file:6cccb5f0a3b3947116a0c0f55d071980d94427ba0d6dad17bc68ead832cc0a8f in /\"],\"Image\":\"\",\"Volumes\":null,\"VolumeDriver\":\"\",\"WorkingDir\":\"\",\"Entrypoint\":null,\"NetworkDisabled\":false,\"MacAddress\":\"\",\"OnBuild\":null,\"Labels\":null},\"docker_version\":\"1.8.2\",\"config\":{\"Hostname\":\"5f8e0e129ff1\",\"Domainname\":\"\",\"User\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"ExposedPorts\":null,\"PublishService\":\"\",\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":null,\"Cmd\":null,\"Image\":\"\",\"Volumes\":null,\"VolumeDriver\":\"\",\"WorkingDir\":\"\",\"Entrypoint\":null,\"NetworkDisabled\":false,\"MacAddress\":\"\",\"OnBuild\":null,\"Labels\":null},\"architecture\":\"amd64\",\"os\":\"linux\",\"Size\":1095501}\n"
      }
   ],
   "signatures": [
      {
         "header": {
            "jwk": {
               "crv": "P-256",
               "kid": "GDNB:2XZW:ROJS:IICZ:NJOI:BFHD:QNI2:SUXB:3T3A:LIAG:JXPX:OIER",
               "kty": "EC",
               "x": "L4f764PuId5WJDPocpZ0Xvilsx1zEgBHG5DskywdktI",
               "y": "aw3o69u9ARuDsNemg5G3lM_LDJO2VCi2zJ_Is1Yv9Fw"
            },
            "alg": "ES256"
         },
         "signature": "G2ffs3lVMgCgbt8nQwqlXmteWCya6mO_Vc4HUIBGb7FnFPL-wbwGOtX5xNcoPqQj17FTVGWnjYKzTyhsIzWfoA",
         "protected": "eyJmb3JtYXRMZW5ndGgiOjMyMDQsImZvcm1hdFRhaWwiOiJDbjAiLCJ0aW1lIjoiMjAxNS0xMC0wNlQyMzoyMjo1MFoifQ"
      },
      {
         "header": {
            "jwk": {
               "crv": "P-256",
               "kid": "GDNB:2XZW:ROJS:IICZ:NJOI:BFHD:QNI2:SUXB:3T3A:LIAG:JXPX:OIER",
               "kty": "EC",
               "x": "L4f764PuId5WJDPocpZ0Xvilsx1zEgBHG5DskywdktI",
               "y": "aw3o69u9ARuDsNemg5G3lM_LDJO2VCi2zJ_Is1Yv9Fw"
            },
            "alg": "ES256"
         },
         "signature": "V2MfUHhRIB7X-ZIB_rmkJqroAZu2RC3eFDHhvbUerT4SvlsoZ6szEVq7xbMaAh6Ot5sNQs_DeyfjQn6l5Aa0wg",
         "protected": "eyJmb3JtYXRMZW5ndGgiOjMyMDQsImZvcm1hdFRhaWwiOiJDbjAiLCJ0aW1lIjoiMjAxNS0xMC0wOFQxNzozMTo0OVoifQ"
      }
   ]
}

Obviously I'm missing something, because our manifests are not the same :)

[TomasTomecek]

Oh, I'm sorry. I should have provide proper reproducer. Here it is:

set -ex
cat >Dockerfile <<EOF
FROM scratch
MAINTAINER ľščťť <mail@example.com> "asd qwe"
EOF
docker build --tag=e .

docker pull registry:2
docker run --net=host --name=reg -d registry:2
docker tag e localhost:5000/e
docker push localhost:5000/e
curl -s http://localhost:5000/v2/e/manifests/latest | egrep -o "MAINTAINER.+\]"
docker stop reg
docker rm reg
docker rmi e localhost:5000/e

My current registry:2 is 24126a56805b with digest sha256:638ece223dd56a62dac03581392eb05266db82309dadc9dbe66c9877ad0c5b6c.

As you'll be able to see, the egrep will show mismatch in encoding.

[RichardScothern]

As per the JSON spec text is encoded as UTF-8 (which is why é and ô are encoded in two bytes as c3 a9 and c3b4 respectively) and angle brackets are represented as unicode codepoints "\u003c" and "\u003e". This is consistent with your first example and consistent across manifests.

What problem is this causing you?

[TomasTomecek]

Aha, didn't know that the brackets are escaped by purpose. So I guess this is fine then if it's your specification.

(the issue I had was I thought that bracket escaping is a bug and I need to workaround that)

Thanks for the pointer!

[TomasTomecek]

Out of curiosity. What is the reason to escape angle brackets and ampersand?

[stevvooe]

@TomasTomecek That is a Go stdlib protection to avoid html injection, I believe. It is described here.