Shortcutting upload of new container

[ruckc]

So I am trying to splice together new container from existing layers in harbor.

I have a requirement to combine a software container by adding a rather large data layer (30GB) to the software container. Given that the software will change due to software changes, library upgrades, etc... we recognize that the software components will evolve much faster than the data, and we want to avoid transferring and storing the rather large amount of data in each new container.

Ideally, the deployment platforms (docker/kubernetes) only have to pull the updated software layers, but not the data layer. Additionally, when our DevOps builds an updated software container, it doesn't have to download the data layer, assemble the new container, and reupload the same 30gb.

So I've got two containers stored, one with the software and all of it's layers, and one with just a single layer with the 30gb of data.

I am pulling down the two manifests/configs of both containers and splicing them together (appending their layers, history, rootfs). I can upload the config, but the manifest fails because the "blob unknown to registry". I've tried the HEAD endpoint /v2/{image}/blobs/{digest} and it returns 404 unknown.

Is there a way for it to recognize it already has the layer blobs from the other images in the project?

[Vad1mo]

Seems like you are already deep down the rabbit hole. Based on the size I guess you are working with some sort of AI models? Did you had a look into similar projects ramalama, CloudNativeAI, kitops. they might already have some solutions related to your problem.

Now regarding your question on the 404, there is too little context to provide you with a qualified answer. Do you do your "splicing" following OCI spec 1.1? OCI referen?

[bupd]

I believe Harbor itself offers layer deduplication, meaning if you upload a layer (such as your large data layer of 30GB) to Harbor once, it can be reused across different images. Just make sure you're correctly referencing the same digest for the data layer in different containers.

BTW it would be helpful. if you can share how you are uploading the config and manifests. manually.

[ruckc]

Background

We are trying to avoid too much specialized tooling for this use case. While some of the containers with data are AI Models, others are just geospatial data that the applications "require" locally. I really just don't want to end up storing 1700 layers of the same 30gb every time the software has a bug fix or dependency upgrade. Also, if we can reuse the same data layer digest then the startup (transfer) time from the container registry to the runtime is significantly reduced.

Overview

So, my rough logic is... given two remote container images localhost/test/cplice-software:latest and localhost/test/cplice-data:latest, and a output image url of localhost/test/cplice:latest.

I have a python script that GET the localhost/v2/{image}/manifests/{tag} and the referenced config digest /v2/{image}/blobs/{digest} for both images. I then essentially just concatenate the data container into the config's history and layer lists. In the manifest, I append the data layer to the manifest's layer list. Then I POST the updated config to localhost/v2/test/cplice/blobs/uploads/?digest={newconfigdigest} and then I try POST localhost/v2/test/cplice/manifests/latest.

I assume I am doing something wrong. I tried HEAD'ing each blob to see if the output image location picks up the blob but it doesn't.

Error

This last manifest POST results the below... i am assuming because the digests aren't shared between images in the repository. I'm also confused because I store the config blob at the new image location just fine, but the error can't find that blob either.

{
  "errors": [
    {
      "code": "MANIFEST_BLOB_UNKNOWN",
      "message": "blob unknown to registry",
      "detail": "sha256:945615621ccc4631801d0ad7423c6d6b3a64b81f2c53b754fc87fc67e0e21f56"
    },
    {
      "code": "MANIFEST_BLOB_UNKNOWN",
      "message": "blob unknown to registry",
      "detail": "sha256:00cfa263d07fd6aac6cfabd1da258894d624252d86433180d08407b4cb9c4c55"
    },
    {
      "code": "MANIFEST_BLOB_UNKNOWN",
      "message": "blob unknown to registry",
      "detail": "sha256:a8ad488006300862f76ff41e1497264808a6ed87c9e71968d285d33d39ad94a2"
    },
    {
      "code": "MANIFEST_BLOB_UNKNOWN",
      "message": "blob unknown to registry",
      "detail": "sha256:0bdd8b92fe601cae450dd8017cadcc515da920d0e56f9f437001ec6192ac4118"
    }
  ]
}

Payloads

Software Image

Manifest

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 654,
    "digest": "sha256:dca58e86e1cd18f168cff1ca44922c44cefbf8d3ff3a1d288b9467178fe99edc"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 92,
      "digest": "sha256:00cfa263d07fd6aac6cfabd1da258894d624252d86433180d08407b4cb9c4c55"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 173,
      "digest": "sha256:a8ad488006300862f76ff41e1497264808a6ed87c9e71968d285d33d39ad94a2"
    }
  ]
}

Config

{
  "architecture": "amd64",
  "config": {
    "Env": [
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ],
    "WorkingDir": "/app"
  },
  "created": "2025-03-15T14:40:04.255784887-04:00",
  "history": [
    {
      "created": "2025-03-15T14:33:44.017663088-04:00",
      "created_by": "WORKDIR /app",
      "comment": "buildkit.dockerfile.v0"
    },
    {
      "created": "2025-03-15T14:40:04.255784887-04:00",
      "created_by": "COPY ./software.txt /app/software.txt # buildkit",
      "comment": "buildkit.dockerfile.v0"
    }
  ],
  "os": "linux",
  "rootfs": {
    "type": "layers",
    "diff_ids": [
      "sha256:29ad9b922879fd6d49153812bb6505afab3ada323425a269e19d65ddb3df66c6",
      "sha256:8f94e8d25d0e76bf341bc79f1751d2ebb643f9d7122241a3f623666370857457"
    ]
  }
}

Data Image

Manifest

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 455,
    "digest": "sha256:c059276ffcc8f304a87f6a33484f0c42d13aa02324b0bbb52ae8c222426324f0"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 168,
      "digest": "sha256:0bdd8b92fe601cae450dd8017cadcc515da920d0e56f9f437001ec6192ac4118"
    }
  ]
}

Config

{
  "architecture": "amd64",
  "config": {
    "Env": [
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ],
    "WorkingDir": "/"
  },
  "created": "2025-03-15T14:34:50.979407054-04:00",
  "history": [
    {
      "created": "2025-03-15T14:34:50.979407054-04:00",
      "created_by": "COPY data.txt /data/data.txt # buildkit",
      "comment": "buildkit.dockerfile.v0"
    }
  ],
  "os": "linux",
  "rootfs": {
    "type": "layers",
    "diff_ids": [
      "sha256:c5fdd9268205ad141335331a6e2c92d88fc66368fd6c655466e437b7c38b6899"
    ]
  }
}

Output Image

Manifest

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 654,
    "digest": "sha256:945615621ccc4631801d0ad7423c6d6b3a64b81f2c53b754fc87fc67e0e21f56"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 92,
      "digest": "sha256:00cfa263d07fd6aac6cfabd1da258894d624252d86433180d08407b4cb9c4c55"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 173,
      "digest": "sha256:a8ad488006300862f76ff41e1497264808a6ed87c9e71968d285d33d39ad94a2"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 168,
      "digest": "sha256:0bdd8b92fe601cae450dd8017cadcc515da920d0e56f9f437001ec6192ac4118"
    }
  ]
}

Config

{
  "architecture": "amd64",
  "config": {
    "Env": [
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ],
    "WorkingDir": "/app"
  },
  "created": "2025-03-15T14:40:04.255784887-04:00",
  "history": [
    {
      "created": "2025-03-15T14:33:44.017663088-04:00",
      "created_by": "WORKDIR /app",
      "comment": "buildkit.dockerfile.v0"
    },
    {
      "created": "2025-03-15T14:40:04.255784887-04:00",
      "created_by": "COPY ./software.txt /app/software.txt # buildkit",
      "comment": "buildkit.dockerfile.v0"
    },
    {
      "created": "2025-03-15T14:34:50.979407054-04:00",
      "created_by": "COPY data.txt /data/data.txt # buildkit",
      "comment": "buildkit.dockerfile.v0"
    }
  ],
  "os": "linux",
  "rootfs": {
    "type": "layers",
    "diff_ids": [
      "sha256:29ad9b922879fd6d49153812bb6505afab3ada323425a269e19d65ddb3df66c6",
      "sha256:8f94e8d25d0e76bf341bc79f1751d2ebb643f9d7122241a3f623666370857457",
      "sha256:c5fdd9268205ad141335331a6e2c92d88fc66368fd6c655466e437b7c38b6899"
    ]
  }
}

[ruckc]

Also, my test code/test case is here https://github.com/ruckc/cplice