nFPM(deb): Empty directory are generated wrongly

[marcauberer]

What happened?

My Goreleaser config contains the following nfpm configuration section:

nfpms:
  - id: publish
    ...
    formats:
      - apk
      - deb
      - rpm
    ...
    contents:
      - src: predefined-services/*
        dst: /usr/lib/compose-generator/predefined-services/
      - src: /usr/bin/compose-generator
        dst: /usr/bin/cg
        type: symlink
      - src: bash_autocomplete.sh
        dst: /etc/bash_completion.d/compose-generator
      - dst: /usr/lib/compose-generator/templates
        type: dir
      - dst: /var/log/compose-generator
        type: dir

Goreleaser does create the Debian packages successfully, but if I try to install them via the apt-get package manager, the installation fails with the following error message:

Unpacking compose-generator (1.6.0) ...
dpkg: error processing archive /var/cache/apt/archives/compose-generator_1.6.0_amd64.deb (--unpack):
 error creating directory './usr/lib/compose-generator/templates': No such file or directory
dpkg-deb: error: paste subprocess was killed by signal (Broken pipe)
Errors were encountered while processing:
 /var/cache/apt/archives/compose-generator_1.6.0_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

How can we reproduce this?

The broken debian package to reproduce the issue can be found here.
The whole Goreleaser configuration file can be found here.
The publish workflow log can be found here (if useful).

goreleaser version

Goreleaser version >= 1.0.0. Presumably due to the upgrade to nFPM 2.9.1

GoReleaser Check

• goreleaser check shows no errors

Search

• I did search for other open and closed issues before opening this.

Code of Conduct

• I agree to follow this project's Code of Conduct

Additional context

No response

[caarlos0]

I tried multiple versions of your package, the last working version was 1.2.1

wget -O 121.deb https://github.com/compose-generator/compose-generator/releases/download/1.2.1/compose-generator_amd64.deb
wget -O 130.deb https://github.com/compose-generator/compose-generator/releases/download/1.3.0/compose-generator_amd64.deb
dpkg -c 121.deb | awk '{print $6}' > 121.txt
dpkg -c 130.deb | awk '{print $6}' > 130.txt
diff 121.txt 130.txt

these are the results:

> ./usr/lib/compose-generator/log
> ./usr/lib/compose-generator/templates
> ./etc/
> ./etc/bash_completion.d/
> ./etc/bash_completion.d/compose-generator
2,5d6
< ./usr/lib/
< ./usr/lib/compose-generator/
< ./usr/lib/compose-generator/templates/
< ./usr/lib/compose-generator/log/
6a8
> ./usr/bin/cg
7a10,11
> ./usr/lib/
> ./usr/lib/compose-generator/
264a269,282
> ./usr/lib/compose-generator/predefined-services/frontend/drupal/
> ./usr/lib/compose-generator/predefined-services/frontend/drupal/README.md
> ./usr/lib/compose-generator/predefined-services/frontend/drupal/config.json
> ./usr/lib/compose-generator/predefined-services/frontend/drupal/service.yml
> ./usr/lib/compose-generator/predefined-services/frontend/drupal/volumes/
> ./usr/lib/compose-generator/predefined-services/frontend/drupal/volumes/drupal-sites/
> ./usr/lib/compose-generator/predefined-services/frontend/drupal/volumes/drupal-sites/default/
> ./usr/lib/compose-generator/predefined-services/frontend/drupal/volumes/drupal-sites/default/default.settings.php
> ./usr/lib/compose-generator/predefined-services/frontend/drupal/volumes/drupal-sites/default/settings.php
> ./usr/lib/compose-generator/predefined-services/frontend/ghost/
> ./usr/lib/compose-generator/predefined-services/frontend/ghost/README.md
> ./usr/lib/compose-generator/predefined-services/frontend/ghost/config.json
> ./usr/lib/compose-generator/predefined-services/frontend/ghost/environment.env
> ./usr/lib/compose-generator/predefined-services/frontend/ghost/service.yml
300a319,322
> ./usr/lib/compose-generator/predefined-services/frontend/mediawiki/
> ./usr/lib/compose-generator/predefined-services/frontend/mediawiki/README.md
> ./usr/lib/compose-generator/predefined-services/frontend/mediawiki/config.json
> ./usr/lib/compose-generator/predefined-services/frontend/mediawiki/service.yml
366d387
< ./usr/bin/cg

My guess is this:

< ./usr/lib/
< ./usr/lib/compose-generator/
< ./usr/lib/compose-generator/templates/

still investigating though

[caarlos0]

the only thing "weird" seems to be the order, actually

sort 121.txt > sorted_121.txt
sort 130.txt > sorted_130.txt
diff sorted_121.txt sorted_130.txt 

> ./etc/
> ./etc/bash_completion.d/
> ./etc/bash_completion.d/compose-generator
7c10
< ./usr/lib/compose-generator/log/
---
> ./usr/lib/compose-generator/log
262a266,279
> ./usr/lib/compose-generator/predefined-services/frontend/drupal/
> ./usr/lib/compose-generator/predefined-services/frontend/drupal/config.json
> ./usr/lib/compose-generator/predefined-services/frontend/drupal/README.md
> ./usr/lib/compose-generator/predefined-services/frontend/drupal/service.yml
> ./usr/lib/compose-generator/predefined-services/frontend/drupal/volumes/
> ./usr/lib/compose-generator/predefined-services/frontend/drupal/volumes/drupal-sites/
> ./usr/lib/compose-generator/predefined-services/frontend/drupal/volumes/drupal-sites/default/
> ./usr/lib/compose-generator/predefined-services/frontend/drupal/volumes/drupal-sites/default/default.settings.php
> ./usr/lib/compose-generator/predefined-services/frontend/drupal/volumes/drupal-sites/default/settings.php
> ./usr/lib/compose-generator/predefined-services/frontend/ghost/
> ./usr/lib/compose-generator/predefined-services/frontend/ghost/config.json
> ./usr/lib/compose-generator/predefined-services/frontend/ghost/environment.env
> ./usr/lib/compose-generator/predefined-services/frontend/ghost/README.md
> ./usr/lib/compose-generator/predefined-services/frontend/ghost/service.yml
298a316,319
> ./usr/lib/compose-generator/predefined-services/frontend/mediawiki/
> ./usr/lib/compose-generator/predefined-services/frontend/mediawiki/config.json
> ./usr/lib/compose-generator/predefined-services/frontend/mediawiki/README.md
> ./usr/lib/compose-generator/predefined-services/frontend/mediawiki/service.yml
366c387
< ./usr/lib/compose-generator/templates/
---
> ./usr/lib/compose-generator/templates

also before dirs had a trailing /

any ideas @erikgeiser?

[caarlos0]

changing your config to

    contents:
      - src: predefined-services/*
        dst: /usr/lib/compose-generator/predefined-services/
      - src: /usr/bin/compose-generator
        dst: /usr/bin/cg
        type: symlink
      - src: bash_autocomplete.sh
        dst: /etc/bash_completion.d/compose-generator
      - dst: /usr/lib/compose-generator
        type: dir
      - dst: /usr/lib/compose-generator/templates
        type: dir
      - dst: /var/log/compose-generator
        type: dir

fixes the issue it seems (as a workaround)

[caarlos0]

pretty sure its the ordering

carlos@darkstar in /tmp/compose 
❯ dpkg -c 121.deb | grep 'usr/lib/compose-generator' | grep -v predefined
drwxr-xr-x 0/0               0 2021-11-08 10:37 ./usr/lib/compose-generator/
drwxr-xr-x 0/0               0 2021-11-08 10:37 ./usr/lib/compose-generator/templates/
drwxr-xr-x 0/0               0 2021-11-08 10:37 ./usr/lib/compose-generator/log/

carlos@darkstar in /tmp/compose 
❯ dpkg -c 130.deb | grep 'usr/lib/compose-generator' | grep -v predefined
drwxr-xr-x root/root         0 2021-11-16 16:35 ./usr/lib/compose-generator/log
drwxr-xr-x root/root         0 2021-11-16 16:35 ./usr/lib/compose-generator/templates
drwxr-xr-x 0/0               0 2021-11-16 16:35 ./usr/lib/compose-generator/

carlos@darkstar in /tmp/compose 
❯ dpkg -c ./compose-generator/dist/compose-generator_amd64.deb | grep 'usr/lib/compose-generator' | grep -v predefined
drwxr-xr-x root/root         0 2021-12-28 10:34 ./usr/lib/compose-generator/
drwxr-xr-x root/root         0 2021-12-28 10:34 ./usr/lib/compose-generator/templates/

[erikgeiser]

I haven't had the time to debug it myself yet but maybe I can give some pointers.

• First the contents are sorted (primarily by destination) after expanding the globs
• Then we loop over all dir entries and create them, with trailing /:
  normalizedName := normalizePath(strings.Trim(file.Destination, "/")) + "/"
• Then we loop over all contents (including dirs) again and create the parent tree, again with trailing /:
  path = normalizePath(path) + "/"
• Then we create the actual content entry if it is not dir, as these are already handled above

This is a bit convoluted but we need to include the parent tree of a dir entry but we cannot create it right away as the next dir entry might specified detailed such as specific permissions for a parent dir of the earlier entry which would have already be written to the archive otherwise.

Now you might say that we sorted the contents before, but I didn't want to rely on that, because it happened in a totally different package and just assuming ordercould be a huge foot gun for those that use it as a library.

[caarlos0]

This is a bit convoluted but we need to include the parent tree of a dir entry but we cannot create it right away as the next dir entry might specified detailed such as specific permissions for a parent dir of the earlier entry which would have already be written to the archive otherwise.

yeah, just found that tout as well, also found the same thing happens on apk, and fixed there too. RPMs don't seem to care about it, though

goreleaser/nfpm#431

[marcauberer]

Thanks for the quick response and the fix pr 👍
I can confirm that the workaround works.