Garbled output when downloading sources - Display of multiple downloads from AUR overlap

[mbwgh]

Affected Version

yay v11.0.1 - libalpm v13.0.1 (before the upgrade)
yay v11.0.2 - libalpm v13.0.1 (after)

Describe the bug

When downloading sources for a full system upgrade, output is garbled, but only during the download phase.

Reproduction Steps

1. $ yay
2. Do a full system upgrade where several sources have to be fetched

Expected behavior

Output for concurrent (if that is it?) downloads should not become mangled and appear as one line per download.

Output

yay -Pg:

{
	"aururl": "https://aur.archlinux.org",
	"buildDir": "/home/mbw/.cache/yay",
	"editor": "",
	"editorflags": "",
	"makepkgbin": "makepkg",
	"makepkgconf": "",
	"pacmanbin": "pacman",
	"pacmanconf": "/etc/pacman.conf",
	"redownload": "no",
	"rebuild": "no",
	"answerclean": "",
	"answerdiff": "",
	"answeredit": "",
	"answerupgrade": "",
	"gitbin": "git",
	"gpgbin": "gpg",
	"gpgflags": "",
	"mflags": "",
	"sortby": "votes",
	"searchby": "name-desc",
	"gitflags": "",
	"removemake": "ask",
	"sudobin": "sudo",
	"sudoflags": "",
	"requestsplitn": 150,
	"sortmode": 0,
	"completionrefreshtime": 7,
	"sudoloop": false,
	"timeupdate": false,
	"devel": false,
	"cleanAfter": false,
	"provides": true,
	"pgpfetch": true,
	"upgrademenu": true,
	"cleanmenu": true,
	"diffmenu": true,
	"editmenu": false,
	"combinedupgrade": false,
	"useask": false,
	"batchinstall": false
}

[Jguer]

This bug (more unimplemented feature) is confirmed. Currently concurrent download just throws all the output to stdout.

Ideas welcome

[mbwgh]

This bug (more unimplemented feature) is confirmed. Currently concurrent download just throws all the output to stdout.

Ideas welcome

I don't speak go, but on a high level I'd maybe try to approach the problem like the following.

There is a maximum of N <= 25 concurrent download threads, often, much less. It should thus be realistic to stack N progress indicators on top of each other and update them inplace (assuming the shell supports this), similar to how docker does this.

In PKGBUILDRepos instead of using a single progress int, I'd try using an an array of progress values or probably directly the string to output, one for each running job. You'd need to keep track of the job number, or maybe it can be queried from the semaphore.
These progress array elements would not need to be synchronized then.

Output would be done only in the main thread (then I don't think you'd even need the mutex?). You can set the shell's vertical cursor position with dedicated escape characters or by shelling out to something like tput, but it may be tricky to make this play nicely with every shell or terminal emulator out there.

[myknbani]

It seems more like a regression. I don't remember this garbling bug as a longtime yay user.

EDIT: I also turned on ParallelDownloads = 5 in /etc/pacman.conf recently, so I'm not sure if that affects things.

[chandujr]

I also turned on ParallelDownloads = 5 in /etc/pacman.conf recently, so I'm not sure if that affects things.

I don't have parallel downloads enabled, but still getting this jumbled log.

[afiorillo]

I found some time to dig into this a bit and can hopefully at least share some insights.

As far as I can tell, this originates in yay when it's actually executing all of the makepkg commands. In downloadPKGBUILDSourceFanout (link) it sends off one goroutine per PKGBUILD and allows up to N concurrent workers (defaulting to number of CPU cores). Each of these eventually constructs a makepkg command (link) and then runs Show which just spews stdout/stderr to the terminal. By this point it's just yay running N concurrent makepkg commands (with the PKGBUILDs downloaded but not the sources).

The console messages such as "Retrieving packages..." and "downloading..." actually come from makepkg -- here namely. According to the Git blame, some changes were made to the cursor positioning in the progress bar about 2 years ago, which would line up with when I think this issue started happening to yay.

I think the easiest option would be to just disable the progress bar. This can be done with the --noprogressbar flag when calling makepkg or by uncommenting the #NoProgressBar line in /etc/pacman.conf. This is just a workaround though and would write that it's downloading the file, though no status information. That said, given how messy it is, I doubt anyone is getting status information currently. The flag could be added to the command builder in yay so as to not affect users' plain pacman usage.

To keep status information I don't see many options. One could be to have yay capture stdout/stderr from the individual makepkg processes, somehow keep those pipes isolated and selectively write the output in a nicer format.

It seems like Pacman does internally support concurrent downloads (hence the ParallelDownloads configuration mentioned above) but the way yay is using it doesn't touch this. yay creates N independent pacman processes, rather than 1 process with N packages. If the install workflow instead bundled all of the makepkg commands into one (somehow, though I'm unsure if makepkg allows this) then that could be an alternative solution.

Edit: of course that isn't the end of the story. Elsewhere in makepkg, sometimes it calls curl instead of uses the custom progress bar inside of Pacman. In particular, download_file pulls the appropriate "download agent," which seems to be different ways of calling curl. When I tested with yay anki, the dependencies of anki were all downloaded with Pacman (and because I modified the pacman conf the progress bars were suppressed), but anki itself was downloaded with that shell script and still spat out curl's progress bar. Interestingly, curl also supports parallel transfers, but I would say it's nearly hopeless to use that here -- yay is making N makepkg processes, each of which makes an independent curl process; to consolidate it into one curl process would be an enormous undertaking.

[Jguer]

Nice research @afiorillo, the garbled output is indeed a consequence of the parallel sourcing of packages and not treating its output correctly. Maybe there is a better way of aggregating the output (the no progress is a nice idea).

maxconcurrentdownloads can be used to limit to one download a time and have a proper display.

It's my opinion now that the concurrent downloads are not that useful (specially in it's current form) and it would be better to remove the feature/option but it hasn't been costly to maintain