This repository has been archived by the owner. It is now read-only.

Cleanup subformulae downloads #17414

Closed
wonbyte opened this Issue Jan 29, 2013 · 24 comments

Comments

Projects
None yet
5 participants
Contributor

wonbyte commented Jan 29, 2013

I just updated to the latest version of git:

(gps ~)$ brew info git
git: stable 1.8.1.2, HEAD
http://git-scm.com
/usr/local/Cellar/git/1.8.1.2 (1267 files, 26M) *
https://github.com/mxcl/homebrew/commits/master/Library/Formula/git.rb
==> Options
--with-blk-sha1
Compile with the block-optimized SHA1 implementation
--with-pcre
Compile with the PCRE library
==> Caveats
The OS X keychain credential helper has been installed to:
/usr/local/bin/git-credential-osxkeychain

The 'contrib' directory has been installed to:
/usr/local/share/git-core/contrib

zsh completion has been installed to:
/usr/local/share/zsh/site-functions

and noticed that running brew cleanup did not remove git-manpages-1.8.1.1.tar.gz. It removed the outdated formula and the htmldocs but left manpages. Is there a reason cleanup left it?

Contributor

adamv commented Jan 29, 2013

Cleanup isn't aware of subformulae

Contributor

wonbyte commented Jan 29, 2013

@adamv why does running:
(gps ~)$ brew cleanup -ns
Would remove: /Library/Caches/Homebrew/distribute-0.6.34.tar.gz
Would remove: /Library/Caches/Homebrew/git-htmldocs-1.8.1.2.tar.gz
Would remove: /Library/Caches/Homebrew/pip-1.2.1.tar.gz

Show git-htmldocs for the latest version I have installed. I thought scrub does not remove installed versions?

Contributor

jacknagel commented Jan 29, 2013

(But it does remove the htmldocs tarball and not the manpages)

Contributor

wonbyte commented Jan 29, 2013

@jacknagel Correct. Removed htmldocs and left manpages.

Contributor

wonbyte commented Jan 29, 2013

(gps ~)$ ls -la /Library/Caches/Homebrew/
total 207016
drwxr-xr-x 33 gps admin 1122 Jan 29 12:32 ./
drwxrwxrwt 13 root admin 442 Jan 29 08:52 ../
drwxr-xr-x 21 gps admin 714 Jan 28 10:04 Formula/
-rw-r--r-- 1 gps admin 164075 Jan 27 15:24 ack-1.96.tgz
-rw-r--r-- 1 gps admin 30691106 Jan 27 15:39 apple-gcc42-4.2.1-5666.3.pkg
-rw-r--r-- 1 gps admin 1927468 Jan 27 15:47 autoconf-2.69.tar.gz
-rw-r--r-- 1 gps admin 2145853 Jan 27 15:48 automake-1.13.1.tar.gz
-rw-r--r-- 1 gps admin 7009201 Jan 27 15:43 bash-4.2.42.tar.gz
-rw-r--r-- 1 gps admin 211899 Jan 27 15:24 bash-completion-1.3.tar.bz2
-rw-r--r-- 1 gps admin 3188400 Jan 27 15:44 curl-7.28.1.tar.gz
-rw-r--r-- 1 gps admin 644358 Jan 27 15:33 distribute-0.6.34.tar.gz
-rw-r--r-- 1 gps admin 655599 Jan 27 15:30 gdbm-1.10.tar.gz
-rw-r--r-- 1 gps admin 4297221 Jan 29 12:31 git-1.8.1.2.tar.gz
-rw-r--r-- 1 gps admin 2051331 Jan 29 12:32 git-htmldocs-1.8.1.2.tar.gz
-rw-r--r-- 1 gps admin 528447 Jan 27 15:46 git-manpages-1.8.1.1.tar.gz
-rw-r--r-- 1 gps admin 528481 Jan 29 12:32 git-manpages-1.8.1.2.tar.gz
-rw-r--r-- 1 gps admin 850772 Jan 28 10:04 libevent-2.0.21.tar.gz
-rw-r--r-- 1 gps admin 439213 Jan 27 15:54 libgpg-error-1.10.tar.bz2
-rw-r--r-- 1 gps admin 624704 Jan 27 15:54 libksba-1.3.0.tar.bz2
-rw-r--r-- 1 gps admin 2632347 Jan 27 15:48 libtool-2.4.2.tar.gz
-rw-r--r-- 1 gps admin 4915203 Jan 27 15:52 libxml2-2.8.0.tar.gz
-rw-r--r-- 1 gps admin 3435907 Jan 27 15:53 libxslt-1.1.28.tar.gz
-rw-r--r-- 1 gps admin 471759 Jan 27 15:51 libyaml-0.1.4.tar.gz
-rw-r--r-- 1 gps admin 3613392 Jan 27 15:24 mercurial-2.4.2.tar.gz
-rw-r--r-- 1 gps admin 4457113 Jan 27 15:48 openssl-1.0.1c.tar.gz
-rw-r--r-- 1 gps admin 102413 Jan 27 15:33 pip-1.2.1.tar.gz
-rw-r--r-- 1 gps admin 226283 Jan 27 15:30 pkg-config-0.27.1.mountainlion.bottle.tar.gz
-rw-r--r-- 1 gps admin 11793433 Jan 27 15:31 python-2.7.3.tar.bz2
-rw-r--r-- 1 gps admin 13781940 Jan 27 15:34 python3-3.3.0.tar.bz2
-rw-r--r-- 1 gps admin 2277926 Jan 27 15:30 readline-6.2.4.tar.gz
-rw-r--r-- 1 gps admin 1846768 Jan 27 15:30 sqlite-3.7.15.1.tar.gz
-rw-r--r-- 1 gps admin 407164 Jan 28 10:04 tmux-1.7.tar.gz
drwxr-xr-x 45 gps admin 1530 Jan 28 08:58 vim--hg/

Contributor

jacknagel commented Jan 29, 2013

Not surprising that this is broken, this is an actual line from cleanup:

if (f and f.version > version) or (ARGV.switch? "s" and (f and (not f.installed?))) or old_bottle
Contributor

jacknagel commented Jan 29, 2013

What @adamv says is true though, formula don't maintain references to subformulae, so that this worked at all was a coincidence.

Contributor

wonbyte commented Jan 29, 2013

@jacknagel So cleanup is just supposed to remove 'git' and not the htmldocs or manpages since they are subformulae. It just happens to be doing that as well, correct? Is it ok to manually remove them from /Library/Cache/Homebrew?

Contributor

jacknagel commented Jan 29, 2013

Yeah. Presumably the regexp that matches filenames also matches the subformulae tarballs.

You can remove anything in the cache, it's just a cache after all.

Contributor

wonbyte commented Jan 29, 2013

@jacknagel Are there any plans to have cleanup remove subformulae?

Contributor

adamv commented Jan 29, 2013

I sketched up a Homebrew2 API that made subformulae explicit in the DSL once

Contributor

adamv commented Jun 12, 2013

I'm working on this in #20212.

Contributor

adamv commented Jun 12, 2013

(If anyone wants to state what they expect the behavior of brew cleanup to be specifically, that would be helpful.)

Contributor

wonbyte commented Jun 12, 2013

I think as @jacknagel said "Presumably the regexp that matches filenames also matches the subformulae tarballs." But the Homebrew devs should also weigh in.

Contributor

adamv commented Jun 12, 2013

I'm a Homebrew dev...but I never use brew cleanup so I don't know how it ought to work.

Contributor

wonbyte commented Jun 12, 2013

I know, I'm just looking for @samueljohn @MikeMcQuaid to weigh in. I think the main issue I experience as a user is that subformulae tarballs are not removed. So when I do a clean up on git after an upgrade the 'manpages" tar ball does not get removed. I have to manually go in and clear the cache.

Owner

MikeMcQuaid commented Jun 13, 2013

I expect brew cleanup to delete all outdated tarballs (including subformulae) from the cache and all non-current (or pinned) versions from the Cellar.

@ghost ghost assigned adamv Sep 12, 2013

Contributor

adamv commented Sep 14, 2013

$ brew cleanup -s android-sdk
Error: No such file or directory - /Users/adamv/homebrew/Cellar/android-sdk

Is this expected when a formula isn't installed but has files in the cache?

Contributor

jacknagel commented Sep 20, 2013

The current implementation of "clean up the cache" isn't very good. This is what it does:

  • For each file in the cache:
    • Try to detect a version from the filename. If this fails, skip to next file.
    • Try to determine a formula name from the filename, by constructing a regexp using the version from the previous step. If this fails, go to the next file.
    • Now try to load the formula for the name. If this fails, go to the next file.
    • If f.version > detected version, remove the file, or
    • if -s was passed and f.installed? is false, remove the file.
    • if bottle_file_outdated?(f, file) is true, remove the file.

This can fail in a lot of ways, mostly because our filename conventions are somewhat inconsistent.

Now that we have resources, and all cached files should begin with "name" or "name--resource_name", we should be able to improve this.

A few complications remain:

  • VCS checkouts are in directories named "name--vcs_name" (and sometimes svn checkouts are "name--svn-HEAD" for some reason?), which overlaps with the resource naming convention.
  • Sometimes file extensions are wrong if the URL is weird, e.g. mysql is currently cached as "mysql-5.6.13.com".
  • Resources without versions (i.e. they can't be detected from the URL and no explicit version is provided) are cached as "name--resource_name-.ext". This also causes problems if the checksum changes.
Contributor

jacknagel commented Sep 20, 2013

Is this expected when a formula isn't installed but has files in the cache?

Currently it doesn't do any cache cleanup when formula arguments are given.

Contributor

adamv commented Sep 20, 2013

(and sometimes svn checkouts are "name--svn-HEAD" for some reason?)

For times when HEAD and one or more of stable & devel are all vcs checkouts. Common when a project has submodules that they don't package in a tarball.

Contributor

jacknagel commented Sep 20, 2013

What I was getting at is that we only do it for svn checkouts, in other cases (git, hg, etc) the checkout is shared by stable and head. Maybe it's not as straightforward to do this with svn?

Contributor

adamv commented Sep 20, 2013

Ah, OK.

My memory of why I set this up gets hazy at this point.

@apjanke apjanke modified the milestone: Clear out Legacy May 3, 2016

Owner

MikeMcQuaid commented Aug 10, 2016

We don't really use subformulae any more and this is broadly encompassed by Homebrew/brew#568.

@MikeMcQuaid MikeMcQuaid locked and limited conversation to collaborators Aug 18, 2016

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.