This project is for the Gentoo portage system to cleanup or organize DISTDIR
Shell
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
bin
etc
zsh
README

README

(C) Martin Väth <martin@mvath.de>
The license of this project is the GNU Public License GPL-2.

This project is for the Gentoo portage system to cleanup or organize DISTDIR.
You can also use it to temporarily enable/disable getdelta or log your fetches.

There is an ebuild for this project in the mv overlay (available by layman).


Detailed description:

There are several scripts in the net or the portage tree which should
cleanup your DISTDIR, for instance eclean from app-portage/gentoolkit.
However, these scripts either remove too much or too less
(because, e.g. they do not reliably follow your useflags).

Moreover, they cannot straightforwardly be used if you want to create
a "minimal" DISTDIR for another system (which e.g. has no internet
connection and limited disc space.)
Also, you can usually not find out easily with such scripts if your DISTDIR
is no longer up-to-date, because e.g. there were silent package bumps.

A solution to all these problems is to use a wrapper over FETCHCOMMAND
to log the files which are actually fetched.

The script trickyfetch of this project is such a wrapper:
To install it, simply copy bin/trickyfetch (with executable permissions)
into a directory from $PATH, and etc/trickyfetch.conf into /etc or into
another directory from $PATH and then define in your
/etc/portage/make.conf (or /etc/make.conf):

FETCHCOMMAND="trickyfetch \"\${URI}\" \"\${DISTDIR}\" \"\${FILE}\""

If you already had set FETCHCOMMAND, you can put this instead into
LOCALFETCHCOMMAND (the trickyfetch script will use this similarly as
portage would use your FETCHCOMMAND; for using getdelta.sh, see below).

This change will not modify your "ordinary" work with portage.
However, the main idea is that now you can cleanup your DISTDIR with
the following steps:

1. Create a subdirectory $DISTDIR/.obsolete and move all regular (i.e. not
   starting with ".") files of $DISTDIR into this subdirectory:
	cd $DISTDIR  (use the correct location of your $DISTDIR, of course)
	mkdir .obsolete
	mv -i -- * .obsolete
2. Make portage fetch all missing files from DISTDIR:
	PORTAGE_CHECKSUM_FILTER='-* size' emerge system world
   If you want to be more careful (and use >=portage-2.2), use instead better
	PORTAGE_CHECKSUM_FILTER='-* size' \
		emerge -feD --with-bdeps=y @system @world

(There is also a tiny bin/obsolete" script in this project which does
exactly the two steps above [with the >=portage-2.2 variant]. You might also
want to copy this script to your path if you use it often.)

That's it: Afterwards, your $DISTDIR directory will contain all required files
and the $DISTDIR/.obsolete subdirectory contains the obsolete files.
All the magic happens in step 2: Instead of actually fetching the files from
the net, the trickyfetch script will just move the files from the
$DISTDIR/.obsolete subdirectory into $DISTDIR.  Of course, files which cannot
be found in that subdirectory will be fetched from the net, as usual.

Optionally, you might want to set the environment variable
	LOCALFETCHCOMMAND=false
in step 2 (or when running the obsolete script):
This way, there is never any access to the net, even if files are missing
or their size/checksum in the Manifest is wrong. (Of course, if you do this
emerge will report corresponding errors and might rename corresponding
files due to checksum errors.)

The trickyfetch script also has a lot of additional features:

(a) It supports app-portage/getdelta and can temporarily switch it off.
(b) It supports logging for the files actually attempted to fetch from the net.
(c) It supports LOCALFETCHCOMMAND to define your own way of fetching data.
(d) It supports wget as a fallback (WGET, WGET_EXTRA_OPTS)
(e) It helps you to keep certain files permanently and "organize" your DISTDIR.
    This is also support by the bin/distlinks script provided by this project.

These points must be explained in more detail:


(a) Support for app-portage/getdelta:

This is optional; support for this can be enabled or disabled
in /etc/trickyfetch.conf

Note that current versions of app-portage/getdelta are broken:
	http://bugs.gentoo.org/show_bug.cgi?id=239439
I recommend the one of the mv overlay which is working.

If you have installed app-portage/getdelta, the trickyfetch script
will use it automatically, i.e. in contrast to the description of
app-portage/getdelta you need not modify anything if you use trickyfetch.
(Provided, of course, that you enabled the support at all in
/etc/trickyfetch.conf).

If want to disable getdelta (even if you enabled the support in
/etc/trickyfetch.conf), either set the environment variable
NODELTA=* (either locally or in your make.conf), or create
the file $DISTDIR/.nodelta
Note that creating/removing that file has the advantage that you can change
"on the fly" (during portage is fetching) whether getdelta should be used
for further packages or not.

The variable NODELTA mentioned above contains a (space-separated)
list of masks; if the name of the file which should be fetched
matches a mask listed in NODELTA, then getdelta is not used.

In addition to NODELTA you can also use EXCLUDE_DELTA:
If EXCLUDE_DELTA contains the name of a file, then the masks
(with the same meaning as in NODELTA) are read from this file.
If EXCLUDE_DELTA contains the name of a directory, the masks are
read from all non-hidden files in this directory or subdirectories.

I recommend to set EXCLUDE_DELTA in your make.conf and to set
NODELTA only locally for specific commands.

If your GENTOO_MIRROR contains more than one mirror, getdelta is not used
if you are fetching from a later mirror than the first: This avoids that
getdelta is used after it already failed for the first mirror.
(This test can be faster if you have eix installed).

If you do not have getdelta installed (or if getdelta is excluded or failed)
a default wget command is used.


(b) Support for logging:

This can be setup in /etc/trickyfetch.conf
See that file for details.


(c) LOCALFETCHCOMMAND support for your own scripts:

If you define LOCALFETCHCOMMAND, this will be called after logging,
but instead of any net access (i.e. getdelta or wget will not be called).
LOCALFETCHCOMMAND takes the role of portage's original FETCHCOMMAND
(and must be set completely analogously if you want to use it).

It is explicitly admissible that LOCALFETCHCOMMAND points to a script
which in turn calls trickyfetch. (To avoid recursion or double logging,
this script should then obviously clear or unset the environment variables
LOCALFETCHCOMMAND and TRICKYLOG.)

Note that you can make LOCALFETCHCOMMAND point to your own script in which
you can in turn call trickyfetch - if you define LOCALFETCHCOMMAND=''
(or unset it) this will not cause a recursion, but causes trickyfetch
to proceed with getdelta.

As mentioned earlier, in the context of e.g. the obsolete script, you might
want to define temporarily
	LOCALFETCHCOMMAND=false
to avoid any internet access when that script is run, but in this case
you might have to manually care later about renamed files by portage.


(d) WGET, WGET_EXTRA_OPTS:

In WGET you can define wget and "standard" options to wget.
Of course, you can use another tool instead of wget; however, this tool
has to support the options -P and -O of wget which are passed immediately
after $WGET.
WGET_EXTRA_OPTS is meant to be set temporarily in special situations
(e.g. when you need to pass options like --no-check-certificate
which you do not want to pass regularly, of course).
Both variables are eval'ed, that is, you can use quoting or even write
a shell function in WGET to handle the option -P/-O and its argument.
Be aware that due to this eval even WGET_EXTRA_OPTS can be misused to
inject arbitrary shell commands into the fetch process!

(e) Keep certain files permanently and "organize" your DISTDIR:

Directories of the form  $DISTDIR/.save* (you can have several of those)
are treated similarly to $DISTDIR/.obsolete by trickyfetch, but with a
lower priority. In other words, trickyfetch works as follows:
It first checks if the file can be found in $DISTDIR/.obsolete,
then it checks if it can be found in some $DISTDIR/.save* (alphabetical order).
The first matching file is moved out of this subdirectory into $DISTDIR.
Only if no such file can be found, the file is fetched from the internet.

Originally, the point here was to move files into $DISTDIR/.save which
you think you need in the near future.
However, this is not a permanent solution, since trickyfetch will move
the files from $DISTDIR/.save to $DISTDIR, once portage needs to fetch them.

For some files, e.g. for fetch-restricted files, you might want a more
permanent solution. This is also supported by trickyfetch:
trickyfetch will always create symlinks to the files in the special directory
$DISTDIR/.restricted
Thus, you can e.g.  move fetch-restricted files in this subdirectory, and
these files need not be fetched by trickyfetch.

A different kind of permanent solution using the $DISTDIR/.save* directories
can be obtained with the bin/distlinks" script.

The usage of this distlinks script is somewhat more complex and explained
in the following. This approach is reasonable, e.g. if you have a
"main machine" from which you want to copy only the relevant portions
of DISTDIR to other machines (e.g. with limited disk space and no
internet connection).

The usage of the bin/distlinks script requires some preparation
(besides copying it into your path, of course):
You must move $DISTDIR one directory level down, e.g. if you originally had
	DISTDIR=/usr/portage/distdir
you should now create a subdirectory like
	DISTDIR=/usr/portage/distdir/portage-distdir
and move the original content of /usr/portage/distdir into that subdirectory.
Do not forget to set the new correct DESTDIR in your make.conf!

The idea is that within the original /usr/portage/distdir (in the following
called $PARENT) you can add further directories like
	$PARENT/overlays
	$PARENT/laptop
	$PARENT/local-projects
into which you can move files from $DISTDIR which should never be deleted.
The purpose of the "distlinks" script is to organize corresponding
$DISTDIR/.save.* subdirectories of symbolic links.
For instance, if you have the subdirectories of the above example, the call
	distlinks (which is the same as distlinks create)
will create the subdirectories
	$DISTDIR/.save.overlays
	$DISTDIR/.save.laptop
	$DISTDIR/.save.local-projects
and fill them with symbolic links to the corresponding files in $PARENT/...
(and by default it will cleanup existing such symbolic links from
$DISTDIR or $DISTDIR/.obsolete).

This way, trickyfetch will use the files from $PARENT (by moving them out
of $DISTDIR/.save.* if necessary), and you can see which of these files
have actually been fetched after the last call of "distlinks create"
by looking at the content of $DISTDIR/.save.*

Note that for efficiency reasons, the trickyfetch script does not treat
subdirectories of $DISTDIR/.save* (or $DISTDIR/.restricted).
However, the "distlinks" script also allows subdirectories.
For instance, you can have
	$PARENT/overlay/local-overlay
	$PARENT/overlay/layman-overlays/mv
The "distlinks" script will create links from $DISTDIR/.save.overlays/*
to all files contained in some of the above directories. It will not warn
about name collisions arising in this way - the first matching name is used.

Note that the symbolic links in $DISTDIR/.save* must be absolute, since the
links should still be valid, if they are moved into $DISTDIR.
The "distlinks" script will create such absolute links
(generating them from the DISTDIR path).

The "distlinks" script can also remove the links and empty $DISTDIR/.save*
subdirectories. Some directories are special for "distlinks":
	$PARENT/empty
	$PARENT/fetch
Their content of these directories is not linked into $DISTDIR/.save.*
The $PARENT/empty directory is created/removed by with appropriate permissions,
so that you can simply "mv" or "cp -a" this directory to create new one.
The $PARENT/fetch directory is meant to be used as storage directory
for live ebuilds. To use it, put into your make.conf something like

ESCM_STORE_DIR=$DISTDIR/../fetch
ECVS_TOP_DIR=$ESCM_STORE_DIR
EBZR_STORE_DIR=$ESCM_STORE_DIR
ECVS_STORE_DIR=$ESCM_STORE_DIR
EGIT_STORE_DIR=$ESCM_STORE_DIR
EHG_STORE_DIR=$ESCM_STORE_DIR
EMTN_STORE_DIR=$ESCM_STORE_DIR
ESVN_STORE_DIR=$ESCM_STORE_DIR

The "distlinks" and "obsolete" script runs slightly faster if eix is installed.

To find out how to use "distlinks" exactly, use
	distlinks -h

To obtain zsh-completion support for the "distlinks" script,
you can copy zsh/_distlinks to a directory of zsh's $fpath