Redesign Aura

[fosskers]

Create a design document in .md format for easy viewing on github. This also allows for code snippets to be added. The main section needed is Functional Requirements, as many of Aura's features were tacked on later because the use cases they involve were not known about at first. As many use cases as possible should be considered before recoding begins.

Related issues:

• Hotedit: --hotedit install files. #208
• Install method: support pkgbase() #213 aura -N #214 Support build directory that is different than the pkg cache directory #203 -M fails to take account of split packages #125 Interactive search/install #114 Aura's temporal directories doesn't work well with AUR packages that require manual downloads. #107 Color output with -S #202
• Package signing: Aura cannot handle signed packages #211 Aura installation fails when Pacman SigLevel = PackageRequired #38
• Help: --help for aura-only operations #111
• Colour: Colours should only be shown when the output is a terminal #102 support pacman’s --color #158 Color output with -S #202
• customizepkg: customizepkg failure & output #93 customizepkg is unsafe #143 Handling of customizepkg #198
• Dependencies: Dependency check failure not verbose enough #215

[fosskers]

There needs to discussion about --hotedit and customizepkg. @dhouck I merged your pull request, and things looked good, but when I got thinking about it, and read this I think forgoing any PKGBUILD editing might be the way to go.

The Problem

Aura's current behaviour is this, assuming --hotedit:
0. Call API to confirm existence of package.

1. Download PKGBUILD.
2. Write to a temporary file.
3. Open it in editor.
4. Read it back as data.
5. Download package sources (PKGBUILD on AUR, .install, etc.)
6. Overwrite that PKGBUILD will the edited one.
7. Check dependencies
8. Call makepkg

With AUR 3.0, dependency information is going to be available via the AUR API, meaning the PKGBUILD will never have to be downloaded in first place. Fewer HTTP calls mean a faster Aura. The process becomes:
0. Call API to confirm existence of package and get its dep info.

1. Check dependencies.
2. Download package sources (PKGBUILD on AUR, .install, etc.)
3. Call makepkg.

As dep checks happen before the PKGBUILD is downloaded, editing the dep fields in the PKGBUILD is pointless. If the user were given the opportunity to alter the deps given by the API, they would also have to alter the PKGBUILD in the same way, or else makepkg would crash.

The Case for --(hot)edit

Some packages require you choose a certain build method by (un)commenting certain sections in build(). xmobar-git is one such example. Which is better:

• Allow the user to use --edit to change the PKGBUILD so they can choose their build style, or;
• Disable all manual editing and tell them to build it manually.

[bb010g]

I would still like a --edit.

[fosskers]

@bb010g Which would allow the user to edit all files in the downloaded tarball? They would have to understand that editing the deps fields wouldn't do anything.

[bb010g]

Just have it edit the metadata from the AUR. (That could go well with something like customizepkg built in, but acting on those fields.)

[fosskers]

I'll have to see what the shape of the metadata is before I can make a call on that.

[dhouck]

For the ABS, wouldn't you still need to read the PKGBUILD to find the makedepends and checkdepends (even if you use the local binary package database for the regular and optdepends)? Given that, would it really be simpler to use the AUR's dependency list instead of reusing the dependency checking code needed for the ABS?

Also, for the current (pre-AUR3) workflow, what's the point of doing steps 1 and 5 separately? It seems like it might be more efficient to do the following, but if this approach is missing something crucial:

0. Call API to confirm existence of package.
1. Download package sources (PKGBUILD on AUR, .install, etc.)
2. Prompt for edits; open relevant files in editor
3. Check dependencies
4. Call `makepkg`

EDIT: Start list at 0 to match ones above

[fosskers]

Question: How much dependency information is stored in the database? Are {make,check}depends excluded?

As for the current flow, you're probably right that doing those things separately is silly. There are redundant http requests, and the size difference between downloading a .tar.gz and just the PKGBUILD is nearly negligible.
The only problem I can see is getting back into a folder once you've untarred it. That is,
with your suggested flow, each tar would be downloaded, opened, and have their PKGBUILD read. Then aura has to back out of that folder, go into another one, and after all dep checks are finalized, go back into the folders in the proper order and call makepkg in there.
So the problem is, right now all tars are downloaded into temp folders that aura's haskell underpinnings delete automatically after building. This gives the following sort of folder hierarchy:

foo1234            # temp name created by haskell
  |--- foo.tar.gz  # the downloaded tarball
  |--- foo/        # the tarball's uncompressed contents
        |--- PKGBUILD
        |--- foo.install

And every folder for every package would look like this. Navigating from folder to folder would take some regexing on package names to find out where to navigate to. Right now, it just does it one at a time. The pre-downloaded PKGBUILD is held internally, dep checks are done, packages are sorted into a nice build order, and then one-by-one their tars are downloaded and built.

Regardless, we don't have to worry about any of this in Aura 2 because of the new API coming, and the libalpm bindings to rely on. I agree with you that the current method is inefficient, but until Aura 2 it probably won't be changed.

[dhouck]

I assume the question at the top of your comment was about libalpm, because AUR 3.0 handles all four types of dependencies and even allows separate dependencies in split packages (which I don't think Aura correctly handles now).

I believe ALPM only records regular and optdepends (which makes sense, given that it's designed for binary packages and therefore doesn't need to know about the others). The ALPM doxygen documentation only gives functions to retrieve these two dependencies (or it puts the makedepends and checkdepends functions somewhere other than where it puts the depends and optdepends ones. Thus, unless it's hiding somewhere I (and Google) haven't seen it, it's undocumented, or you want to drop ABS support entirely, you'll need to get these fields from PKGBUILDs directly when using the ABS.

Because of the ABS, the updated workflow you posted might not always be usable and some variant of the existing one might be necessary, at least sometimes, so I have another question about it. Would there be a problem with having a single temp directory with all the packages under it, like below:

Aura1234            # temp name created by haskell
  |--- foo.tar.gz  # the downloaded tarball
  |--- foo/        # the tarball's uncompressed contents
        |--- PKGBUILD
        |--- foo.install
  |--- depend.tar.gz # foo depends on depend
  |--- depend/
        |--- PKGBUILD
        |--- depend.rc

[rmarquis]

Just a few remarks, since I had also time to think about these issues for another project:

Support for customizepkg is dropped, as AUR 3.0 provides dependency information via its API.
Support for --hotedit dropped. As dependency checking is done through the API and not the PKGBUILDs, editing PKGBUILDs could cause a discrepancy between aura and makepkg. If you need fine-grained control over how the package is being built, building it manually with makepkg is recommended. If you're trying to fix a broken package, please contact the maintainer.

I fully agree with the above. There shouldn't be any reason to modify dependencies in the PKGBUILDs in the first place. And if there is one, an additional PKGBUILD should probably be uploaded in the AUR instead of doing "magic" locally (that's the very reason I believe the idea behind customizepkg is silly in the first place).

An --edit flag should be kept for special cases like xmobar-git, although any modification to the deps fields would be ignored.

Just have it edit the metadata from the AUR.

The .AURINFO metadata is only useful to make the complete info available in the RPC interface in a secure way (and adding split package support in the AUR too), so "editing the metadata" would be the equivalent of editing the Json response.

Then, it's possible to edit the json info on the fly, if you keep one info call per package or dependency. But Aura should use the multiinfo RPC call instead, because it is way faster as it requires only one call per level of dependencies, no matter what is the total number of dependencies in the transaction.

Of course it would also theoretically be possible to edit the returned multiinfo answer, but that's would probably a nightmare for the user in practice. Better forget the customizationpkg thing entirely, it's not worth it (and silly in the first place imho).

or you want to drop ABS support entirely, you'll need to get these fields from PKGBUILDs directly when using the ABS.

I know some core pacman developers are interested in integrating ABS closely with pacman itself (read it some time ago on the ML). I haven't seen any patch for it yet, but I know this will eventually happen. Maybe in two weeks, maybe in two years, I don't know. So it's up to the aura dev to decide if it is worth it integrating it now, or if the extra work and time would be better invested in some other area of aura.

[fosskers]

Premise: If you know enough about package building to know you want different dependencies, you're smart enough to build them manually while hand-editing the PKGBUILD.
Conclusion: No dependency editing will be supported. If dependencies are broken, the maintainers need to be contacted and the AUR versions updated. Things like --hotedit are actually letting packages lag behind if users are able to edit everything themselves on the fly, because it doesn't force the maintainers to keep up to date packages. And if the users really want custom deps that badly, they can building manually as stated above.

An --edit flag will be used purely for editing build behaviour. Any dep changes will be ignored by Aura (but notably, not by makepkg).

---

And yes, one multiinfo call for everything will be quite nice.

[dhouck]

Since you've decided to keep hotediting, is there a reason not to also keep customizepkg support? If it's made clear that customizepkg also cannot be used for Aura's metainformation useful for changing building behavior when you want to make the same changes to a new version's files. Often, you want to keep the same changes whenever you reinstall the package. With customizepkg (or at least customizepkg-scripting or customizepkg-patching; I don't know about the original), this can be handled automatically; without, you need to re-run Aura with --hotedit (and -k, but I always use that) in order to see exactly what changes you made and to make them again.

I'll agree that changing dependencies isn't that useful, but there is a case where "contact the maintainer" or "release a clone package" aren't better options: adding optional dependencies to the depends array (because you want the dependency, and installing optional dependencies still doesn't work well whether or not you use --asdeps even if things are getting better). In this case, it's possible to install the optdepend --asdeps manually, then to make the changes to the package and just not worry about Aura not knowing. This would be a bit hacky, but would still work even without Aura tracking package metadata changes.

[rmarquis]

Often, you want to keep the same changes whenever you reinstall the package.

What prevents you from uploading your own version in the AUR? I don't see in which way that wouldn't be a better option than patching again and again a package locally.

[dhouck]

In general: when somebody wants to make a change to an package and they believe their change would only benefit a small number of people (with others neutral or disliking the change), it is generally not worth uploading the package to the AUR. It's a minor hassle for the maintainer (and doesn't get rid of the problem of "patching again and again a package locally"), and if only a small number of people would prefer their version to a version already in the AUR or the official repositories, it doesn't give much benefit. Often, this change may only be useful to that one person (one example might be somebody who wants to configure dwm but still have it managed by a package manager, but I'm not sure if this example works because I don't personally use dwm or any of the other several tiling window managers that require recompilation for a configuration change ).

The specific package that prompted my earlier comment: There's a package in the community repo called terminus-font. There are also four versions of it in the AUR with different configurations, only one of which has much support and none of which are up-to-date. I want a fifth configuration which is almost but not quite identical to one of the ones without much support already in the AUR, so uploading the changed package to the AUR (and then maintaining it) probably wouldn't be worth the effort and might make things harder for anybody looking through the AUR (because they now have more nearly identical choices and they're unlikely to want my particular changes).

[fosskers]

Could you not just keep a local PKGBUILD that you bump the version of every once and a while?
customizepkg has in the past introduced a number of problems (which to use? how to catch its errors? is it secure at all?) Making aura handle customizepkg properly is probably much more work than the payoff would be.

You mention avoiding confusing AUR users with multiple versions on the AUR. I agree with this. We've arrived at one of the fundamental problems of the AUR: it's filled with garbage. Vast expanses of 0-vote, out-of-date, orphaned packages. Is there any system in place to have these removed?

In the mean time, aura -A{i,s} show maintenance status and votes. That's about the only tool we have right now for telling what packages are legitimate.

[rmarquis]

You mention avoiding confusing AUR users with multiple versions on the AUR. I agree with this.

If the modified version is useful for someone, chances are that it is useful for another one at least. If the pkgbuild is kept up-to-date and has a logical name (software-mydiff), I don't really see why it shouldn't be uploaded in the AUR. The fact that it is popular or not is just a side effect.

We've arrived at one of the fundamental problems of the AUR: it's filled with garbage. Vast expanses of 0-vote, out-of-date, orphaned packages. Is there any system in place to have these removed?

Unmaintained or obsolete packages are indeed a big issue. There have been some discussion in the forums and on the aur-dev mailing list, but nothing is set on stone yet.

When I read the following, I really hope the entire AUR content get wiped and starts again on a clean basis:

There are thousands of packages using old PKGBUILD syntax in aur. 34.54% of all aur packages will stop working at pacman 4.2 release. If you also count packages that are plain broken, duplications, outdated and so on, I would not be supriced if the total number is close to 50%,

Seriously, the aurpocalypse is going to be fun.

[fosskers]

When I read the following, I really hope the entire AUR content get wiped and starts again on a clean basis.

I would support that.

[dhouck]

Am I to understand that you don't want to implement customizepkg support, but you wouldn't object to its existing if somebody else were to write it? Or do you expect that it would cause too many headaches even if it was well written initially?

Also, what about a more generic hooks mechanism? ALPM will probably get hooks at some point, but it might be useful to have AUR, ABS, or building specific hooks. They could be used in general to implement customizepkg support, namcap support, etc. It seems like that could be a useful feature to consider when doing a redesign even without the recent conversation.

[fosskers]

I think it's a very good idea (thinking for future version of Aura as well, moved to other distros) to implement some sort of plugin/hooks system so that the build process can be highly customized.

[fosskers]

@dhouck I've made three proposals on how the plugin system could be implemented. What do you think?

[flamusdiu]

@fosskers

Option 3 gives the most flexibility though the CON (many deps) could really cause havoc with the build system.

Option 2 I think is good middle ground and works somewhat like PKGBUILDs now (different areas you can write and such.

Option 1 Too complexed really and honestly Option 2 can give just as much flexibility as 1.

[fosskers]

could really cause havoc with the build system.

How exactly? One other thing I thought of was that non-Haskell plugins wouldn't be forced to compute things via the Aura Monad (in particular, via the Settings held within it). That said, the settings could always be passed to the plugin in the JSON, and they would have to obey it.

...good middle ground...

Unfortunately I've never coded with Haskell actors before and I believe for me personally this would be the hardest to implement. Another pro that hasn't been added yet is the opposite of the above: Operations could be guaranteed to run through the Aura Monad.

Too complex really...

I don't know if I agree, but the whole "pre-built version users can't configure" thing is a big con, I think.

[bb010g]

Too complex really...

I don't know if I agree, but the whole "pre-built version users can't configure" thing is a big con, I think.

We could just do something like aura-contrib for all the good ones.

[fosskers]

That applies to any of the options.

[fosskers]

I'm considering going ahead with Option 1 for the following reasons:

• AuraConf.hs replaces pacman.conf, and becomes a place to define all relevant file paths (package cache, etc.) and build hooks, finally giving us a conf file.
• Distribution of hooks is simple via Hackage packages.
• Turning hooks on/off is just a matter of removing them from whatever hooks list they were a part of:

hooks :: AuraHooks
hooks = defaultHooks

becomes:

import Aura.Hooks.Customizepkg

hooks :: AuraHooks
hooks = defaultHooks { pkgbuildHooks = customizepkgHook : defaultPkgbuildHooks }

• Aura's default behaviour just becomes a matter of defining default hooks.
• Aura can be heavily customized for use by other Distros by ripping out all the default hooks and supplying new ones specific to that distro. The core of Aura then becomes a hook manager and dependency resolver.

[fosskers]

Let me elaborate:

As xmonad, we would have a .hs conf file, where you could define things like:

pkgCache :: Text
pkgCache = "/var/cache/pkgs/"

but it would probably be more "hands off" for the average user to have:

import Linux.Aura.Hooks.Arch (archPkgCache, ...)

pkgCache :: Text
pkgCache = archPkgCache

The user can, of course, change this at will by setting a different string. They know their conf file is valid if it typechecks. Saving their change, they run aura --recompile to rebuild aura in a cabal sandbox and replace the binary in /usr/bin/ or wherever.

Within Aura, all it ever checks is pkgCache, assuming it has been set properly in the conf file (which it will have been, if the whole program compiles).

Q: How will the user know what the actual contents of the conf file (paths, etc.) are if they're mostly aliases to things defined in Hooks?

A: An option like aura --conf that pretty-prints all the relevant info.

[flamusdiu]

Just for a sane check, might want to add a aura --confi-check (or something) so people could make sure the config check before running aura and having it fail out for some reason due to conf checks. If a user does not run the config-check manually before running aura and a failure happens, then a forced check should be run which could also out put the --config flag as well.

[fosskers]

Dyre might support that (if we decide to use it).
2014/07/28 18:05 "flamusdiu" notifications@github.com:

Just for a sane check, might want to add a aura --confi-check (or
something) so people could make sure the config check before running aura
and having it fail out for some reason due to conf checks. If a user does
not run the config-check manually before running aura and a failure
happens, then a forced check should be run which could also out put the
--config flag as well.

—
Reply to this email directly or view it on GitHub
#217 (comment).

[AphonicChaos]

It was a runtime error, so I'm not sure what kind of checking could have prevented it. If it helps make things clearer, I can try to reproduce the scenario this weekend. Another alternative might be the plugin package, ghc's api, or hint. I'll update this post with links within the hour (heading out the door at the moment).

[fosskers]

http://aura.readthedocs.org/en/latest/

[AphonicChaos]

Shiny. Now we just need a better TOC. I'd say some reformating is also in
order. Out of town this weekend but I might tackle that if no one else does
some time after.
On Jul 29, 2014 8:34 PM, "Colin Woodbury" notifications@github.com wrote:

http://aura.readthedocs.org/en/latest/

—
Reply to this email directly or view it on GitHub
#217 (comment).

[fosskers]

I'll be going through and updating some things pretty soon.

[boomshroom]

I would be ok without hotedit as long as we get makepkg functionality in aura. I sometimes use hotedit to completely change what package I'm installing just so that aura can install aur dependancies and make/install the package.

[flying-sheep]

How’s it going?

Did you address #153 in the redesign already?

[fosskers]

@flying-sheep Luckily that will be hook dependent and not part of the Aura core itself.

[joca-bt]

Suggestion: option to pass arguments directly to makepkg. The first example that comes to mind is to allow passing --skippgpcheck to makepkg.

[fosskers]

@joca-bt This is often requested, and would be handled by the Arch-specific hooks of Aura 2.

[atondwal]

Linux.Arch.Aur.* and Linux.Debian.Rep is very misleading. There's no reason aura shouldn't work unmodified on, say, Arch BSD, Arch Hurd, or even Debian kFreeBSD, is there?

[joca-bt]

Cool, thanks!

[fosskers]

@atondwal I didn't know there were non-Linux versions of Arch. Provided they have pacman and makepkg installed, then yeah, aura should work unmodified.

[atondwal]

Maybe we could have names like Distribution.Arch and Distribution.Debian for pacman/dpkg stuff respectively?

[fosskers]

Or take an Aura topname?

[bb010g]

I'd be for just Aura.

[fosskers]

Where a hook library would be: Aura.DistroName.

[flamusdiu]

+1

[bb010g]

#332 brought up an interesting point. Currently, pacman uses both Q and D operations for local database operations (queries & changes) and R for local package removal, and uses S for remote package addition and database queries, T for a mix of remote and local queries (dependency testing for wanted remote packages on the local system), and U for local package addition. V is used for version testing. We'll have to find some way to unify all these into top level operations. I'd suggest a model of two operations per hook: e.g. for the main system package manager, S for remote and Q for local. pacman's R, D, & U would then become sub-operations of Q and pacman's T would become a sub-operation of S. I'd imagine v (née V) would become like h and give info on the parent hook, leaving V for global aura version info.

[fosskers]

My propositions for renaming top-level ops is covered in the Doc. I like the idea of relegating the other ones to subops of the top-level ones.

[bb010g]

The main issue with something like a global R operation is that you can have different packages from different hooks that have the same name. What happens then? An interactive disambiguation would be my first thought, but you can't easily script that. My next thought is a sort of namespacing, and that is what the two operations per hook pattern would provide. Also, on the topic of scripting, it would be nice to have a long, unique variant of each operator (as to provide consistency across setups in scripts). E.g. on an Arch system, -S would also be --alpm and -A would be --aur. A is unique here, as it depends on -L (--alpm-local) for local package management and thus is an exception to the two operations per hook pattern. However, something like --linuxbrew would need a --linuxbrew-local.

[fosskers]

I have been flirting with the idea of multiple Hook sets on one machine (which you seem to be implying here). As pacman does now with haskell packages, could it be the domain of Aura to interface to all package managers on a user's system?

Then they make calls like:

• aura -S firefox <- the default, with no git-like command.
• aura cabal -{S,L} lens <- Still install/query the same way.
• aura npm -{S,L} grunt
• aura pip -{S,L} django

Some general -Ss function could search through all Hook sets and provide multiple sources for the same package.

aura -Ss lens then gives:

haskell-core/haskell-lens 1.x.x: blah blah blah
cabal/lens 1.x.x: blah blah blah

[flamusdiu]

@fosskers that would probably make the day of people who use all these different package manager and would be very useful.

[fosskers]

Is it easier for someone to do aura cabal -S lens or cabal install lens? I'm thinking the later. That said, I think this functionality would really shine during querying.

Output like aura -Ss lens above with local packages would also be very convenient:

> aura -Ls lens
haskell-core/haskell-lens 1.x.x: blah blah blah
cabal/lens 1.x.x: blah blah blah

So that you can see what packages, from any manager, are installed on your system.

[bb010g]

Relevant conversation on the Gitter on July 27th and 28th. The direction is shifting away from "support _ALL_ the things" to "support ALPM and the AUR well".

[fosskers]

Design then may be informed by a proper port of libalpm.