debootstap fails with repository published by aptly #49

Closed
smira opened this Issue May 12, 2014 · 3 comments

Comments

Projects
None yet
2 participants
@smira
Member

smira commented May 12, 2014

Split from #44.

@dupuy reports:

I was trying to use debootstrap with an aptly published repository and getting completely obscure errors during the package validation stage:

/usr/sbin/debootstrap: 626: /usr/sbin/debootstrap: arithmetic expression: expecting EOF: "2255928 + 97b1f7e0e3188ba4acf665c3c31d0c2cca5af2665b6b634357ba90bbc0487341 69658"

(debootstrap tries to parse the package information in its shell script, and the spurious spaces threw it off).

@dupuy

This comment has been minimized.

Show comment
Hide comment
@dupuy

dupuy May 13, 2014

My comment in #44 was only intended to thank you (and to create a pointer for anyone using debootstrap with aptly to know that they needed the fix for that problem) - there are no other issues for debootstrap related to spaces in folder names beyond what you have already fixed.

I have, however, discovered another problem for debootstrap, which is that when working with an aptly-published repo, its dependency resolution is not working - debootstrap is selecting nearly every package in the distribution.

The source of the problem is that the hairy mess of shell with Perl snippets that is debootstrap expects a Package: line as the first one in each stanza in the Packages files, and aptly currently puts an Origin: line before it.

The debootstrap bug is in the pkgdetails_perl function's Perl snippet, where the code that attempts to determine if it is currently looking at a relevant package stanza is incredibly stupid, and will use the package name from the previous stanza if the Package: line is not the first one, causing the dependencies of each stanza following a package to be added to the package set. After a few iterations of that, nearly every package in the repository is included, and this generally causes debootstrap to fail for one reason or another.

After fixing that, I had the converse problem, which was that the debootstrap --variant code was not including crucial packages (with "Priority: important", "Priority: required", or "Build-Essential: yes", depending on the release and debootstrap options); again the problem was that the logic implicitly assumed that the Package: line would be the first one in each stanza (I didn't even bother to analyze the problem with the code, but just put in a crude hack to ignore Origin: lines).

The bugs here are not really aptly's, but as there are probably other programs with equally stupid logic, and it will surely take quite some time for any debootstrap fix to be accepted and propagate downstream, it would probably be wise for aptly to put the Package: line first in the stanzas when it is writing Packages files.

The following is my fix for debootstrap (on a Ubuntu 12.04 system). I don't know when I will have the time to submit this upstream to Canonical or the Debian maintainers, if anybody else reading this wants to do so, please add a comment with the issue / bug / ticket information and link.

The first part of the patch is the crude hack to ignore Origin: lines entirely (so that the one aptly puts preceding the Package: line doesn't prevent the debootstrap --variant code from properly picking up packages) and is not a general fix, although it is otherwise harmless. The second part is more general, and as long as the Package: line precedes the Depends: line, it should work (it also speeds up the dependency computation slightly by running the grep operator against the package list only for Package: lines rather than every line).

--- /usr/share/debootstrap/functions.ORIG   2014-04-25 15:31:10.000000000 +0000
+++ /usr/share/debootstrap/functions    2014-05-13 16:01:47.574129501 +0000
@@ -1064,6 +1064,7 @@
 while (<STDIN>) {
    chomp;
    next if (/^ /);
+   next if (/^Origin:/);
    if (/^([^:]*:)\s*(.*)$/) {
        $f = lc($1); $v = $2;
        if ($f eq "package:") {
@@ -1130,20 +1130,20 @@
    if (/^Package: (.*)$/) {
        $pkg = $1;
        if ($pkg ne $prevpkg) {
            for my $d (@d) {
                print "$d\n";
            }
        }
        $prevpkg = $1;
        @d = ();
+       $in = 1 if (grep {$_ eq $pkg} @ARGV);
    }
-   $in = 1 if (grep {$_ eq $pkg} @ARGV);
    $in = 0 if (/^$/);
    if ($in and (/^Depends: (.*)$/ or /^Pre-Depends: (.*)$/)) {
        for $d (split /\s*,\s*/, $1) {
            $d =~ s/\s*[|].*$//;
            $d =~ s/\s*[(].*[)]\s*//;
            push @d, $d;
        }
    }
 }

dupuy commented May 13, 2014

My comment in #44 was only intended to thank you (and to create a pointer for anyone using debootstrap with aptly to know that they needed the fix for that problem) - there are no other issues for debootstrap related to spaces in folder names beyond what you have already fixed.

I have, however, discovered another problem for debootstrap, which is that when working with an aptly-published repo, its dependency resolution is not working - debootstrap is selecting nearly every package in the distribution.

The source of the problem is that the hairy mess of shell with Perl snippets that is debootstrap expects a Package: line as the first one in each stanza in the Packages files, and aptly currently puts an Origin: line before it.

The debootstrap bug is in the pkgdetails_perl function's Perl snippet, where the code that attempts to determine if it is currently looking at a relevant package stanza is incredibly stupid, and will use the package name from the previous stanza if the Package: line is not the first one, causing the dependencies of each stanza following a package to be added to the package set. After a few iterations of that, nearly every package in the repository is included, and this generally causes debootstrap to fail for one reason or another.

After fixing that, I had the converse problem, which was that the debootstrap --variant code was not including crucial packages (with "Priority: important", "Priority: required", or "Build-Essential: yes", depending on the release and debootstrap options); again the problem was that the logic implicitly assumed that the Package: line would be the first one in each stanza (I didn't even bother to analyze the problem with the code, but just put in a crude hack to ignore Origin: lines).

The bugs here are not really aptly's, but as there are probably other programs with equally stupid logic, and it will surely take quite some time for any debootstrap fix to be accepted and propagate downstream, it would probably be wise for aptly to put the Package: line first in the stanzas when it is writing Packages files.

The following is my fix for debootstrap (on a Ubuntu 12.04 system). I don't know when I will have the time to submit this upstream to Canonical or the Debian maintainers, if anybody else reading this wants to do so, please add a comment with the issue / bug / ticket information and link.

The first part of the patch is the crude hack to ignore Origin: lines entirely (so that the one aptly puts preceding the Package: line doesn't prevent the debootstrap --variant code from properly picking up packages) and is not a general fix, although it is otherwise harmless. The second part is more general, and as long as the Package: line precedes the Depends: line, it should work (it also speeds up the dependency computation slightly by running the grep operator against the package list only for Package: lines rather than every line).

--- /usr/share/debootstrap/functions.ORIG   2014-04-25 15:31:10.000000000 +0000
+++ /usr/share/debootstrap/functions    2014-05-13 16:01:47.574129501 +0000
@@ -1064,6 +1064,7 @@
 while (<STDIN>) {
    chomp;
    next if (/^ /);
+   next if (/^Origin:/);
    if (/^([^:]*:)\s*(.*)$/) {
        $f = lc($1); $v = $2;
        if ($f eq "package:") {
@@ -1130,20 +1130,20 @@
    if (/^Package: (.*)$/) {
        $pkg = $1;
        if ($pkg ne $prevpkg) {
            for my $d (@d) {
                print "$d\n";
            }
        }
        $prevpkg = $1;
        @d = ();
+       $in = 1 if (grep {$_ eq $pkg} @ARGV);
    }
-   $in = 1 if (grep {$_ eq $pkg} @ARGV);
    $in = 0 if (/^$/);
    if ($in and (/^Depends: (.*)$/ or /^Pre-Depends: (.*)$/)) {
        for $d (split /\s*,\s*/, $1) {
            $d =~ s/\s*[|].*$//;
            $d =~ s/\s*[(].*[)]\s*//;
            push @d, $d;
        }
    }
 }
@smira

This comment has been minimized.

Show comment
Hide comment
@smira

smira May 29, 2014

Member

aptly now generates Package: lines first in index files. I hope that helps with debootstrap.

Member

smira commented May 29, 2014

aptly now generates Package: lines first in index files. I hope that helps with debootstrap.

@smira

This comment has been minimized.

Show comment
Hide comment
@smira

smira May 30, 2014

Member

Closing this. @dupuy, please reopen if there are more problems with order. I can make better implementation if that would be required.

Member

smira commented May 30, 2014

Closing this. @dupuy, please reopen if there are more problems with order. I can make better implementation if that would be required.

@smira smira closed this May 30, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment