Can't get manifest plugin to work correctly (w/ Drone 1.0RC1)

[djmaze]

(cross-posted from discourse)

I want to generate a docker manifest in my pipeline. As I am using auto-tagging with the platform-specific images, the manifest should generate those versions (e.g. 0.9.3 / 0.9) as well.

Since this is not properly documented with the manifest plugin, I've tried it this way:

steps:
  - name: manifest
    image: plugins/manifest
    settings:
      target: my/image:${DRONE_TAG}
      template: my/image:${DRONE_TAG}-ARCH
      username:
        from_secret: docker_username
      password:
        from_secret: docker_password
      platforms:
        - linux/amd64
        - linux/arm

But it seems Drone won't escape the variables inside the plugin settings.

My second approach was to use a manifest template with the spec and auto_tag options, so the config looks like this:

steps:
  - name: manifest
    image: plugins/manifest
    settings:
      spec: manifest.tmpl
      auto_tag: true
      username:
        from_secret: docker_username
      password:
        from_secret: docker_password
      platforms:
        - linux/amd64
        - linux/arm

This is not properly documented as well, so I took the template I found in the plugin's repository and adjusted it slightly for my image name. Using that, I get the following output in the manifest step:

2018/11/17 00:19:10 pushing by spec
time="2018-11-17T00:19:10Z" level=fatal msg="Can't unmarshal YAML file \"/drone/src/manifest-014888304\": yaml: mapping values are not allowed in this context" 
2018/11/17 00:19:10 exit status 1

What is the right way to do this?

[djmaze]

Well, I switched from using plugins/manifest to plugins/manifest:1, and now the spec approach works.

Either way, the documentation should probably include a manifest template example.

[deitch]

@djmaze do you mind sharing how you did it? I follow this pattern in a lot of places, built some really "interesting" Makefiles to make it work, and now would like to do this in drone if I can.

Basically, I almost always follow the pattern:

1. When I have binaries, I compile them to dist/bin/<binary_name>-<arch> (for non-compiled like Ruby or Node, it is irrelevant)
2. I have a Dockerfile.<arch> which is unique to that arch (sometimes it is the FROM that changes, sometimes the COPY, etc.)
3. I build and push images with <org>/<repo>:<version>-<arch>, which there are multiple versions (I almost always include the git hash, short or long, since tags are free, sometimes latest as well).
4. I push a manifest for <org>/<repo>:<version> (for multiple version) that references the deployed <org>/<repo>:<version>.

Sounds like you figured it out in this plugin?

[deitch]

Or is it that the manifest plugin only pushes the manifest, and assumes the images already exist, which is what the docker plugin is there for? Is that how you use it?

[tboerger]

Are you using trimPrefix within your template? I found out that trimPrefix haven't been present in latest since it was using the drone template lib, now it's available again but got swapped the parameters: {{trimPrefix "v" build.tag}}

[tboerger]

Or is it that the manifest plugin only pushes the manifest, and assumes the images already exist, which is what the docker plugin is there for? Is that how you use it?

The docker plugin is used to build and push the images, the manifest plugin just defines the manifest and pushs it to the registry.

[deitch]

Thanks @tboerger. I need to hop over to that repo and ask a question about my build pattern there. It looks like this cleanly handles the last stage (very similar to Phil Estes' manifest-tool, which I still use). The first stage is part of core drone, which leaves just the middle stages, all part of the docker plugin.

I saw you opened that PR on "adding docs". Since @djmaze referenced the spec: manifest.tmpl, are there docs on how to use the template?

[tboerger]

very similar to Phil Estes' manifest-tool, which I still use

This plugin is using exactly that tool to push the manifests...

I saw you opened that PR on "adding docs". Since @djmaze referenced the spec: manifest.tmpl, are there docs on how to use the template?

There is still no proper documentation for it. The template will be processed by the https://github.com/aymerick/raymond engine and handed over to manifest-tool. How the template could look like you can already see on most drone-plugins repos within the docker/ folder.

[deitch]

This plugin is using exactly that tool to push the manifests...

Too funny! I believe he has wanted to retire it in favour of docker manifest, but last I checked, it still was missing some functionality.

There is still no proper documentation for it

Ah, it is just using the manifest yml that manifest-tool can accept, but giving it pre-processing? OK, got it.

I will hop onto discourse to see about the rest of the pattern.

Thanks for your assistance.

[deitch]

Done. https://discourse.drone.io/t/compile-binaries-publish-multiple-docker-images-one-per-architecture/3851

[bradrydzewski]

@tboerger do we need to tag a new release so that plugins/manifest:1 is replaced with the newer code? I just ran into this issue here and am trying to figure out the best way to resolve.

It looks like the manifest auto tag logic may not be creating tags for major and minor versions. For example, in dockerhub I see plugins/manifest:1.1.0, however I do not see plugins/manifest:1.1 and plugins/manifest:1 looks old.

[tboerger]

I don't really get why it has not pushed the manifest for 1, it's using the same code as the docker plugin: https://github.com/drone-plugins/drone-manifest/blob/master/tagging/tagging_test.go

[tboerger]

AFAIK this issue should be resolved, if you still encounter this issue please ping on this issue.

[dessalines]

I'm also stuck on this, that spec link is gone so I have no idea how to generate one.

We also use a separate drone builder for amd64, and arm images, but docker manifest create --amend can handle pushing one after the other. Not sure how to do this in a way that can read the current tag, and run some simple docker manifest create and docker manifest push commands....

[dessalines]

Finally got it working!

  - name: push to docker manifest
    image: plugins/manifest
    settings: 
      username:
        from_secret: docker_username
      password:
        from_secret: docker_password
      target: "my_name/my_repo:${DRONE_TAG}"
      template: "my_name/my_repo:${DRONE_TAG}-OS-ARCH"
      platforms:
        - linux/amd64
        - linux/arm64
      ignore_missing: true
    when:
      ref:
      - refs/tags/*