No recursive download with databases other than in gzip

[PedroHLC]

I'm using databases with .tar.zst, before that, I used .tar.xz. I never understood why I couldn't use recursive and now on 0.20 I get this tiny message:

> repoctl --debug down -r xfce4-git-meta
error: gzip: invalid header

Database(s): https://lonewolf.pedrohlc.com/chaotic-aur/x86_64/chaotic-aur.db.tar.zst
repoctl config: https://github.com/chaotic-aur/infra/blob/main/home/main-builder/.config/repoctl/config.toml

[PedroHLC]

Also thanks man, repoctl has been serving me for more than a year already. And today chaotic-aur has more than 7k users monthly. That would be way harder without your tool.

[cassava]

Hey @PedroHLC, glad to hear that repoctl is useful to you! :-)

I must admit that recursive download didn't actually initially work. But I got the kinks worked out and that has been part of the 0.20 release. See issue #33 for more on that.

If you are up for trying out a pre-release commit, I want to push a commit that should solve the problem. Would be nice to hear if that works for you. I wasn't able to reproduce the issue yet, unfortunately.

[PedroHLC]

😕 I was trying from c1a7475, I was able to reproduce in my personal computer like this:

cd /tmp
repo-add example.db.tar.zst
echo 'repo = "/tmp/example.db.tar.zst"' > ~/.config/repoctl/config.toml
repoctl down -r osdlyrics-git

[cassava]

Alright, with the most recent version from develop should be significantly better.

First, building is a lot simple since moving to Go modules:

git clone -b devel https://github.com/cassava/repoctl.git
cd repoctl
go install ./...

And here's what I did to test your use-case.

export repodir=$(mktemp -d)
echo "using repodir=$repodir"
cd $repodir
repo-add example.db.tar.zst
echo "repo = \"$repodir/example.db.tar.zst\"" > $repodir/config.toml
REPOCTL_CONFIG="$repodir/config.toml" repoctl down -r osdlyrics-git

(There's another update in the pipeline that will let you see more package metadata, kind of like with pacman -Qi.)

[cassava]

@PedroHLC Have you had a chance to try out the current develop branch?

[PedroHLC]

I had, and the problem still persists for me. As I build/install from my own PKGBUILD, I'll be doing more tests and compiling more info to handle you a proper bug report or find what I'm doing wrong.

[PedroHLC]

😄 I have no clue what I'm doing, but with or without Arch's go flags, with go install or go build and later installing the executable.
It all ends with error: gzip: invalid header and no other verbosity. 😢
Commit is 1102d3c.
Running your test example snippet.

[cassava]

@PedroHLC
I'm having some trouble reproducing the issue, even with older versions of repoctl, such as v0.18.
I wonder if this might be related to #44.

With the most recent version of repoctl, you can run repoctl query osdlyrics-git to get all the metadata on the package.
Does that work? What happens when you wget or curl the Snapshot URL? Perhaps it's not in the GZ format and that's why it's not working.

[PedroHLC]

that works:

[cassava]

I'd really love to fix this issue, but as of now I've been unable to reproduce it with devel, v0.20, or v0.19.
Here's the script I used for testing:

#!/bin/bash
#
# This file tests repoctl functionality with the current devel branch.
set -e -x

version="${1-devel}"
dbname="example.db.tar.zst"

# 1. Create a temporary directory where we will do our tests.
export testdir=$(mktemp -d)
cd "${testdir}"

# 2. Build the desired version of repoctl.
export GOPATH="${testdir}"
(
    mkdir -p src/github.com/cassava
    cd src/github.com/cassava
    git clone git@github.com:cassava/repoctl.git
    cd repoctl
    git checkout "${version}"
    go install ./...
)
test -x bin/repoctl

# 3. Create a repository where we will put our database and create an
# initial empty database.
mkdir repo
dbpath="${testdir}/repo/${dbname}"
repo-add "${dbpath}"
test -f "${dbpath}"

# 4. Write a configuration for repoctl.
confpath="${testdir}/config.toml"
bin/repoctl new -c "${confpath}" config "${dbpath}"
test -f "${confpath}"
bin/repoctl version

# 5. Use said configuration to recursively download a package.
mkdir abs
(
    cd abs
    REPOCTL_CONFIG="${confpath}" ../bin/repoctl down -r xfce4-git-meta
)
test -f "${testdir}/abs/thunar-git/PKGBUILD"

And here are the results: https://asciinema.org/a/RYr7oS1UMhxx75vKZPeTgVTv1
(That link is valid for about 7 days supposedly. So after that, see the attachments below.)

Attachments:

• asciinema-recording.txt
• test-repoctl.sh.txt

If the problem persists with you it would be helpful to get the output of the test-repoctl.sh script
when run on your machine.

[PedroHLC]

Hi @cassava, I hadn't time to test it yet.
But, would you mind testing with my buildings of repoctl?
From our testbenchs the only thing I can think could interfere, is the building I'm always building within clean chroots (actually, systemd-nspawn) and not go install (I don't let anything touch my rootfs, sorry).

Here's the latest:
http://lonewolf-builder.duckdns.org/chaotic-aur/x86_64/repoctl-devel-git-0.20.r15.gf280958-1-x86_64.pkg.tar.zst

[cassava]

That's very commendable, that you sandbox all foreign code. :-)
I tried your build with your config and your database, and I still can't reproduce the problem.

I suspect it has something to do with the network or the AUR API.
According to the AUR documentation, you can make up to 4000 requests a day per IP address.
I'll see if I can get some better error messages integrated into repoctl at the point of failure.
Maybe then we can see why it prints the error: error: gzip: invalid header

[cassava]

@PedroHLC I finally was able to reproduce the problem on my machine.
The latest commit should resolve the issue. Let me know if that is the case!

[PedroHLC]

Worked flawlessly with c32b18d