Invalid JSON Manifest for Helm Charts in ECR

[rpf3]

Confirm by changing [ ] to [x] below to ensure that it's a bug:

• I've gone through the User Guide and the API reference
• I've searched for previous similar issues and didn't find any solution

Describe the bug
The JSON object returned by ecr batch-get-image is missing the top level "mediaType" property when operating on Helm charts. I noticed this by trying to retag both a Helm chart and a Docker image using this AWS guide. When I inspected the manifest JSON, I saw that the Helm chart was missing that property. If I then manually added it with a value matching the "mediaType" in the "config" object I was able to call ecr put-image without issue.

SDK version number

aws-cli/2.2.21 Python/3.8.8 Linux/5.4.72-microsoft-standard-WSL2 exe/x86_64.ubuntu.20 prompt/off

Platform/OS/Hardware/Device

WSL 2

To Reproduce (observed behavior)

1. Push a Helm chart to an ECR repository.
2. Follow the steps here using batch-get-image to save the manifest.
3. Try to retag the Helm chart using put-image.
4. Error returned:

Invalid parameter at 'ImageManifest' failed to satisfy constraint: 'Invalid JSON syntax'

Expected behavior
The manifest json should be able to be pushed back to ECR using ecr put-image.

[kdaily]

Hi @rpf3,

Sorry to hear you're having a problem. I looked at the API specification for the request to put-image, and it states for the imageManifestMediaType property:

imageManifestMediaType
The media type of the image manifest. If you push an image manifest that does not contain the mediaType field, you must specify the imageManifestMediaType in the request.

Here are the docs specific to pushing a Helm chart:

https://docs.aws.amazon.com/AmazonECR/latest/userguide/push-oci-artifact.html

I don't see anything calling out the issues you're encountering. Unfortunately, this looks like a service issue and is not specific to the AWS CLI. I can let the service team know and get back to you on this issue if they respond, but you could also request assistance through AWS support, or post on AWS re:Post and get a faster resolution.

In order to pass along details to the ECR team, having complete debug logs for these operations would be helpful. You can add --debug to your command to get these, but please redact any sensitive information.

[kdaily]

Since this is a service issue, I'm going to transfer this issue to aws/aws-sdk repository.

[rpf3]

I've attached the debug log but I think the relevant new line is:

botocore.errorfactory.InvalidParameterException: An error occurred (InvalidParameterException) when calling the PutImage operation: Invalid parameter at 'ImageManifest' failed to satisfy constraint: 'Invalid JSON syntax'

debug.log

However if I update the put-image request to use the --image-manifest-media-type flag with a value of "application/vnd.oci.image.manifest.v1+json" it will work successfully.

So I guess the question is whether or not the Helm chart should have a mediaType property in the response from batch-get-image. The JSON for the imageManifest for a Docker image has the following structure:

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 1234,
    "digest": ""
  },
  "layers": []
}

While the response from a Helm chart is:

{
  "schemaVersion": 2,
  "config": {
    "mediaType": "application/vnd.cncf.helm.config.v1+json",
    "digest": "",
    "size": 193
  },
  "layers": []
}

[kdaily]

Thanks for the debug and the details! I'll review and see if I need to loop in the ECR team.

[tim-finnigan]

P72321100

[tim-finnigan]

We heard back from a member of the ECR team who said that the value of imageManifest is the string content of manifest file you pushed to ECR using the PutImage API and ECR does not change it.

How did you push the helm chart manifest the very first time? The expectation is that push used the --image-manifest-media-type flag.

This behavior is already documented here as was previously mentioned:

The media type of the image manifest. If you push an image manifest that does not contain the mediaType field, you must specify the imageManifestMediaType in the request.

So based on the above this does not seem to be a service API issue.

[rpf3]

I first am authenticating to with ECR by running aws ecr get-login-password and then piping the result to helm registry login. Then I publish the chart to ECR with a helm push "{chart_name}" "oci://{registry_name}" command.

Based on your answer, it sounds like this needs to be resolved in the Helm CLI?

[tim-finnigan]

Based on your answer, it sounds like this needs to be resolved in the Helm CLI?

That may be the case - you could try opening an issue with Helm for further feedback.

[rpf3]

@tim-finnigan I've reach out to the Helm team as suggested and they responded that they are following the OCI spec. And reading back on this original thread I keep coming back to my original question of why the response from batch-get-image for a Docker container contains a top level mediaType property while the response for a Helm chart does not. The fact that the Helm response is missing this top level mediaType means that, in accordance with your docs, we need to supply the imageManifestMediaType in the request.

[tim-finnigan]

Thanks for following up. For any issue with the BatchGetDocumentation we recommend reaching out through the Provide feedback link on that page.

[github-actions]

This issue is now closed.

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.