reprepro cannot handle ddeb artifacts

[nuclearsandwich]

The debian toolchain for artful and bionic has started producing ddeb files for ROS packages.

Here's one for cpp_common. The builds run fine until the import_package job is triggered which fails with

13:27:14 Unknown file type: '93418f6e79db403d1bb2d5e731cf5498 104028 debug optional ros-melodic-cpp-common-dbgsym_0.6.8-0bionic.20180126.212400_amd64.ddeb', assuming source format...
13:27:14 Warning: File 'ros-melodic-cpp-common-dbgsym_0.6.8-0bionic.20180126.212400_amd64.ddeb' looks like source but does not start with 'ros-melodic-cpp-common_'!
13:27:14 'ros-melodic-cpp-common-dbgsym_0.6.8-0bionic.20180126.212400_amd64.ddeb' looks like part of an source package, but no dsc file listed in the .changes file!
13:27:14 There have been errors!

There's a circa 2013 reprepro bug with a patch and info but it doesn't appear that anything from that thread made it into reprepro. A grep of the codebase doesn't yield any mention of ddeb packages.
There is a fork of reprepro at https://github.com/profitbricks/reprepro which last year introduced changes to support ddeb files on top of additional changes.

This is also tracked on Launchpad as LP#799899

Still to be pinned down is a description of the changes introduced into the debian toolchain in bionic. Currently we can observe these differences but don't have a way to account for all of them. There is this discussion about compatibility between debian and ubuntu dbgsym packaging.

Some resolution for this is quite urgent as we cannot proceed with Melodic preparations until this is resolved.

Ideas for potential resolutions

1. Suppressing ddeb package creation
2. Renaming -dbgsym.ddeb packages -dbgsym.deb
3. Deleting ddeb packages during import before invoking reprepro
4. Switching to a forked reprepro that supports ddebs
5. The 🍄 ☁️ nuclear option is, of course, switching away from reprepro in favor of aptly but I think that widens the scope well beyond the urgency of this issue.

cc @tfoote as the resident reprepro expert and @clalancette as the melodic leader.

[dirk-thomas]

Same for artful: http://build.ros.org/job/Mbin_uA64__cpp_common__ubuntu_artful_amd64__binary/

[nuclearsandwich]

Same for artful

Thanks, updated the issue body.

[tfoote]

We had a related discussion at: ros-infrastructure/bloom#32 where we decided to stick with the default behaviors for the OS. It would be great to keep the debug symbol package generation but find a way to not push it into the repos so that we can reenable it easily in the future when/if we upgrade/replace reprepro.

Definitely aptly would be great, but also is likely to be the most work and out of scope for a short term fix.

I believe that there might even be a simpler approach for 3) which is to just remove references to the ddeb from the changes file. Then the importer won't know about them since it uses the changes file to import everything.

Format: 1.8
Date: Fri, 26 Jan 2018 13:24:00 -0800
Source: ros-melodic-cpp-common
Binary: ros-melodic-cpp-common
Architecture: amd64
Version: 0.6.8-0bionic.20180126.212400
Distribution: bionic
Urgency: high
Maintainer: Dirk Thomas <dthomas@osrfoundation.org>
Changed-By: Dirk Thomas <dthomas@osrfoundation.org>
Description:
 ros-melodic-cpp-common - cpp_common contains C++ code for doing things that are not necess
Changes:
 ros-melodic-cpp-common (0.6.8-0bionic.20180126.212400) bionic; urgency=high
 .
   * Append timestamp when binarydeb was built.
Checksums-Sha1:
 eec82d395023b02290e84db7e36fbb0e13044510 104028 ros-melodic-cpp-common-dbgsym_0.6.8-0bionic.20180126.212400_amd64.ddeb
 1b0da7533e862b801fd2e7b58fa396c0664084ac 14504 ros-melodic-cpp-common_0.6.8-0bionic.20180126.212400_amd64.buildinfo
 4966eae67cb7d4a96dc10eae2f8cc2ec0eca605b 18928 ros-melodic-cpp-common_0.6.8-0bionic.20180126.212400_amd64.deb
Checksums-Sha256:
 3cfd4de62a83abec6a0b073bdc1de709da8cf8f0349386f829334c636738ad29 104028 ros-melodic-cpp-common-dbgsym_0.6.8-0bionic.20180126.212400_amd64.ddeb
 35a30cc7a6fe0cbc8e334e3a2e2984017b60b3387fa98d8d5a7280ea295eb132 14504 ros-melodic-cpp-common_0.6.8-0bionic.20180126.212400_amd64.buildinfo
 f06ad8cd0f8ef842afc4187c592040abbdb5b4facbc2b94f0740e4c3c8fce478 18928 ros-melodic-cpp-common_0.6.8-0bionic.20180126.212400_amd64.deb
Files:
 93418f6e79db403d1bb2d5e731cf5498 104028 debug optional ros-melodic-cpp-common-dbgsym_0.6.8-0bionic.20180126.212400_amd64.ddeb
 862aa2ab7635022e28b8b47598f66778 14504 misc extra ros-melodic-cpp-common_0.6.8-0bionic.20180126.212400_amd64.buildinfo
 43e32bdc86b51e8ee15f96329c99bd40 18928 misc extra ros-melodic-cpp-common_0.6.8-0bionic.20180126.212400_amd64.deb

Adding a sed -i '/*.ddeb$/d' *.changes just before the import will likely make it pass and be easily removable in the future if we have a simpler solution.

[nuclearsandwich]

One thing that doesn't make sense to me yet. How is the ddeb file getting transferred to the reprepro host when this code here, paired with this template section (itself an invocation of this snippet).

Shouldn't only the .deb and .changes file be moved? Is it possible that the sourceFiles logic is being treated as a regexp rather than a glob?

[nuclearsandwich]

Synthesizing my findings with @tfoote's the ddeb file isn't getting transferred and the error is triggered seemingly by its presence in the changes file.

[tfoote]

That definitely could be that we were reading the error wrong. It's failing because we aren't copying the file over not that it has an unsupported extension. So we should patch it to try uploading the ddebs too and see if reprepro is happy with that. And if that doesn't work we can go the other way and strip the ddeb references from the changes file.

[nuclearsandwich]

It's failing because we aren't copying the file over not that it has an unsupported extension. So we should patch it to try uploading the ddebs too and see if reprepro is happy with that.

For posterity: we tried, same failure

# BEGIN SECTION: import debian package
15:37:02 + export PYTHONPATH=/home/jenkins-agent/workspace/Mrel_import-package/reprepro-updater/src:
15:37:02 + python -u /home/jenkins-agent/workspace/Mrel_import-package/reprepro-updater/scripts/include_folder.py --folder /var/repos/ubuntu/building/queue/artful/Mbin_uA64__cpp_common__ubuntu_artful_amd64__binary__2 --package ros-melodic-cpp-common --delete-folder --commit --invalidate
15:37:02 running command ['reprepro', '-b', '/var/repos/ubuntu/building', '-T', 'deb', '-V', 'removefilter', u'artful', u'Package (== ros-melodic-cpp-common ), Architecture (== amd64 )']
15:37:03 running ['reprepro', '-v', '-b', '/var/repos/ubuntu/building', 'deleteunreferenced']
15:37:03 running command ['reprepro', '-v', '-b', '/var/repos/ubuntu/building', 'deleteunreferenced']
15:37:03 Importing package: ros-melodic-cpp-common:0.6.8-0artful.20180126.233346:artful:amd64
15:37:03 running command ['reprepro', '-v', '-b', '/var/repos/ubuntu/building', '-S', 'misc', 'include', u'artful', '/var/repos/ubuntu/building/queue/artful/Mbin_uA64__cpp_common__ubuntu_artful_amd64__binary__2/ros-melodic-cpp-common_0.6.8-0artful.20180126.233346_amd64.changes']
15:37:04 Unknown file type: 'a94b1e1361a24653a10a02d89c1ce000 101438 debug extra ros-melodic-cpp-common-dbgsym_0.6.8-0artful.20180126.233346_amd64.ddeb', assuming source format...
15:37:04 Warning: File 'ros-melodic-cpp-common-dbgsym_0.6.8-0artful.20180126.233346_amd64.ddeb' looks like source but does not start with 'ros-melodic-cpp-common_'!
15:37:04 'ros-melodic-cpp-common-dbgsym_0.6.8-0artful.20180126.233346_amd64.ddeb' looks like part of an source package, but no dsc file listed in the .changes file!
15:37:04 There have been errors!
15:37:04 Execution of [['reprepro', '-v', '-b', '/var/repos/ubuntu/building', '-S', 'misc', 'include', u'artful', '/var/repos/ubuntu/building/queue/artful/Mbin_uA64__cpp_common__ubuntu_artful_amd64__binary__2/ros-melodic-cpp-common_0.6.8-0artful.20180126.233346_amd64.changes']] Failed: Command '['reprepro', '-v', '-b', '/var/repos/ubuntu/building', '-S', 'misc', 'include', u'artful', '/var/repos/ubuntu/building/queue/artful/Mbin_uA64__cpp_common__ubuntu_artful_amd64__binary__2/ros-melodic-cpp-common_0.6.8-0artful.20180126.233346_amd64.changes']' returned non-zero exit status 255
15:37:04 Build step 'Execute shell' marked build as failure
15:37:04 [description-setter] Could not determine description.
15:37:04 Email was triggered for: Failure - Any
15:37:04 Sending email for trigger: Failure - Any
15:37:04 Sending email to: clalancette+buildfarm@osrfoundation.org ros-buildfarm-melodic@googlegroups.com
15:37:04 Finished: FAILURE

[tfoote]

Approximate approach 3 implemented in ros-infrastructure/reprepro-updater#55 hopefully there will be a 4 or 5 in the future. But closing this for now as that's a long term enhancment and this as a bug is resolved.

[nuclearsandwich]

Is there anything that makes a ddeb a ddeb other than the file extension? Can we rename it to a plain old deb modify the artifact data. As it stands now we'll have no debug packages for Melodic.

[tfoote]

I don't know of anything different besides the filename. But I would hope there's something different to justify the different name, but that's quite possibly wishful thinking. A deb is actually just a tarball anyway.