Cleanup subformulae downloads

[wonbyte]

I just updated to the latest version of git:

(gps ~)$ brew info git
git: stable 1.8.1.2, HEAD
http://git-scm.com
/usr/local/Cellar/git/1.8.1.2 (1267 files, 26M) *
https://github.com/mxcl/homebrew/commits/master/Library/Formula/git.rb
==> Options
--with-blk-sha1
Compile with the block-optimized SHA1 implementation
--with-pcre
Compile with the PCRE library
==> Caveats
The OS X keychain credential helper has been installed to:
/usr/local/bin/git-credential-osxkeychain

The 'contrib' directory has been installed to:
/usr/local/share/git-core/contrib

zsh completion has been installed to:
/usr/local/share/zsh/site-functions

and noticed that running brew cleanup did not remove git-manpages-1.8.1.1.tar.gz. It removed the outdated formula and the htmldocs but left manpages. Is there a reason cleanup left it?

[adamv]

Cleanup isn't aware of subformulae

[wonbyte]

@adamv why does running:
(gps ~)$ brew cleanup -ns
Would remove: /Library/Caches/Homebrew/distribute-0.6.34.tar.gz
Would remove: /Library/Caches/Homebrew/git-htmldocs-1.8.1.2.tar.gz
Would remove: /Library/Caches/Homebrew/pip-1.2.1.tar.gz

Show git-htmldocs for the latest version I have installed. I thought scrub does not remove installed versions?

[jacknagel]

(But it does remove the htmldocs tarball and not the manpages)

[wonbyte]

@jacknagel Correct. Removed htmldocs and left manpages.

[wonbyte]

(gps ~)$ ls -la /Library/Caches/Homebrew/
total 207016
drwxr-xr-x 33 gps admin 1122 Jan 29 12:32 ./
drwxrwxrwt 13 root admin 442 Jan 29 08:52 ../
drwxr-xr-x 21 gps admin 714 Jan 28 10:04 Formula/
-rw-r--r-- 1 gps admin 164075 Jan 27 15:24 ack-1.96.tgz
-rw-r--r-- 1 gps admin 30691106 Jan 27 15:39 apple-gcc42-4.2.1-5666.3.pkg
-rw-r--r-- 1 gps admin 1927468 Jan 27 15:47 autoconf-2.69.tar.gz
-rw-r--r-- 1 gps admin 2145853 Jan 27 15:48 automake-1.13.1.tar.gz
-rw-r--r-- 1 gps admin 7009201 Jan 27 15:43 bash-4.2.42.tar.gz
-rw-r--r-- 1 gps admin 211899 Jan 27 15:24 bash-completion-1.3.tar.bz2
-rw-r--r-- 1 gps admin 3188400 Jan 27 15:44 curl-7.28.1.tar.gz
-rw-r--r-- 1 gps admin 644358 Jan 27 15:33 distribute-0.6.34.tar.gz
-rw-r--r-- 1 gps admin 655599 Jan 27 15:30 gdbm-1.10.tar.gz
-rw-r--r-- 1 gps admin 4297221 Jan 29 12:31 git-1.8.1.2.tar.gz
-rw-r--r-- 1 gps admin 2051331 Jan 29 12:32 git-htmldocs-1.8.1.2.tar.gz
-rw-r--r-- 1 gps admin 528447 Jan 27 15:46 git-manpages-1.8.1.1.tar.gz
-rw-r--r-- 1 gps admin 528481 Jan 29 12:32 git-manpages-1.8.1.2.tar.gz
-rw-r--r-- 1 gps admin 850772 Jan 28 10:04 libevent-2.0.21.tar.gz
-rw-r--r-- 1 gps admin 439213 Jan 27 15:54 libgpg-error-1.10.tar.bz2
-rw-r--r-- 1 gps admin 624704 Jan 27 15:54 libksba-1.3.0.tar.bz2
-rw-r--r-- 1 gps admin 2632347 Jan 27 15:48 libtool-2.4.2.tar.gz
-rw-r--r-- 1 gps admin 4915203 Jan 27 15:52 libxml2-2.8.0.tar.gz
-rw-r--r-- 1 gps admin 3435907 Jan 27 15:53 libxslt-1.1.28.tar.gz
-rw-r--r-- 1 gps admin 471759 Jan 27 15:51 libyaml-0.1.4.tar.gz
-rw-r--r-- 1 gps admin 3613392 Jan 27 15:24 mercurial-2.4.2.tar.gz
-rw-r--r-- 1 gps admin 4457113 Jan 27 15:48 openssl-1.0.1c.tar.gz
-rw-r--r-- 1 gps admin 102413 Jan 27 15:33 pip-1.2.1.tar.gz
-rw-r--r-- 1 gps admin 226283 Jan 27 15:30 pkg-config-0.27.1.mountainlion.bottle.tar.gz
-rw-r--r-- 1 gps admin 11793433 Jan 27 15:31 python-2.7.3.tar.bz2
-rw-r--r-- 1 gps admin 13781940 Jan 27 15:34 python3-3.3.0.tar.bz2
-rw-r--r-- 1 gps admin 2277926 Jan 27 15:30 readline-6.2.4.tar.gz
-rw-r--r-- 1 gps admin 1846768 Jan 27 15:30 sqlite-3.7.15.1.tar.gz
-rw-r--r-- 1 gps admin 407164 Jan 28 10:04 tmux-1.7.tar.gz
drwxr-xr-x 45 gps admin 1530 Jan 28 08:58 vim--hg/

[jacknagel]

Not surprising that this is broken, this is an actual line from cleanup:

if (f and f.version > version) or (ARGV.switch? "s" and (f and (not f.installed?))) or old_bottle

[jacknagel]

What @adamv says is true though, formula don't maintain references to subformulae, so that this worked at all was a coincidence.

[wonbyte]

@jacknagel So cleanup is just supposed to remove 'git' and not the htmldocs or manpages since they are subformulae. It just happens to be doing that as well, correct? Is it ok to manually remove them from /Library/Cache/Homebrew?

[jacknagel]

Yeah. Presumably the regexp that matches filenames also matches the subformulae tarballs.

You can remove anything in the cache, it's just a cache after all.

[wonbyte]

@jacknagel Are there any plans to have cleanup remove subformulae?

[adamv]

I sketched up a Homebrew2 API that made subformulae explicit in the DSL once

[adamv]

I'm working on this in #20212.

[adamv]

(If anyone wants to state what they expect the behavior of brew cleanup to be specifically, that would be helpful.)

[wonbyte]

I think as @jacknagel said "Presumably the regexp that matches filenames also matches the subformulae tarballs." But the Homebrew devs should also weigh in.

[adamv]

I'm a Homebrew dev...but I never use brew cleanup so I don't know how it ought to work.

[wonbyte]

I know, I'm just looking for @samueljohn @MikeMcQuaid to weigh in. I think the main issue I experience as a user is that subformulae tarballs are not removed. So when I do a clean up on git after an upgrade the 'manpages" tar ball does not get removed. I have to manually go in and clear the cache.

[MikeMcQuaid]

I expect brew cleanup to delete all outdated tarballs (including subformulae) from the cache and all non-current (or pinned) versions from the Cellar.

[adamv]

$ brew cleanup -s android-sdk
Error: No such file or directory - /Users/adamv/homebrew/Cellar/android-sdk

Is this expected when a formula isn't installed but has files in the cache?

[jacknagel]

The current implementation of "clean up the cache" isn't very good. This is what it does:

• For each file in the cache:
  • Try to detect a version from the filename. If this fails, skip to next file.
  • Try to determine a formula name from the filename, by constructing a regexp using the version from the previous step. If this fails, go to the next file.
  • Now try to load the formula for the name. If this fails, go to the next file.
  • If f.version > detected version, remove the file, or
  • if -s was passed and f.installed? is false, remove the file.
  • if bottle_file_outdated?(f, file) is true, remove the file.

This can fail in a lot of ways, mostly because our filename conventions are somewhat inconsistent.

Now that we have resources, and all cached files should begin with "name" or "name--resource_name", we should be able to improve this.

A few complications remain:

• VCS checkouts are in directories named "name--vcs_name" (and sometimes svn checkouts are "name--svn-HEAD" for some reason?), which overlaps with the resource naming convention.
• Sometimes file extensions are wrong if the URL is weird, e.g. mysql is currently cached as "mysql-5.6.13.com".
• Resources without versions (i.e. they can't be detected from the URL and no explicit version is provided) are cached as "name--resource_name-.ext". This also causes problems if the checksum changes.

[jacknagel]

Is this expected when a formula isn't installed but has files in the cache?

Currently it doesn't do any cache cleanup when formula arguments are given.

[adamv]

(and sometimes svn checkouts are "name--svn-HEAD" for some reason?)

For times when HEAD and one or more of stable & devel are all vcs checkouts. Common when a project has submodules that they don't package in a tarball.

[jacknagel]

What I was getting at is that we only do it for svn checkouts, in other cases (git, hg, etc) the checkout is shared by stable and head. Maybe it's not as straightforward to do this with svn?

[adamv]

Ah, OK.

My memory of why I set this up gets hazy at this point.

[MikeMcQuaid]

We don't really use subformulae any more and this is broadly encompassed by Homebrew/brew#568.