roopwn

#! /usr/bin/env python3

'''"roopwn": "Running out of Pacman wrapper names"'''

"""TODO:
    * maybe use pyalpm instead of Pacman CLI calls?
    * somehow need to combine Pacman --downloadonly and --print operations; the
        --print option does not ask questions about resolving providers but 
        --downloadonly does, and the answer may not be what --print assumes;
        thus it may indicate packages that weren't actually downloaded.
    * /lib/libc.so.6: version `GLIBC_2.16' not found (required by git-upload-
        pack)
    * handle the dependencies of the package that create the /usr/lib/ or
        /lib/ symlink, or any symlink
    * need to notify about conflicts before downloading the (wrong) package
    * temporary packages: remove them instead of upgrading them
    * -C --downgrade [aura] use package cache as source
    * Can CLI options be ripped from "pycman" stuff in "pyalpm" package?
    * Show URL for package info: AUR search or package page; etc
    * Specify flags applying to certain packages or sets of packages; eg
        check file deps by default, except for a specific few
    * Allow source priority to be overridden on CLI, eg --priority gnome-
        unstable,staging; after that use Pacman's configured priorities
    * Avoid finding duplicate libraries due to different symlinks (/lib/ and
        /usr/lib/)
    * run namcap etc automatically before installing packages; but not for
        "trusted" packages. This would be a completely different kind of
        trust to the signed packages trust; you should be able to trust the
        signature of a package to avoid malicious packages yet not trust it
        for namcap testing.
    * if a file dep is not met, could check it in package file db; if found
        in opt deps then that would be okay (but I don't use the package file
        db). Also could use that DB to search for packages providing a
        missing dep
    * Option to force unlock sync DB
    * Some sort of friendly search and install mode

Metapac:
    * generate an optional dependency package based on optdepends info (group
        identical explanations, except for missing explanations)
    * build a dummy package that installs nothing but provides a given
        package name

Package builder:
    * register different sources: binary and source repositories
    * source repositories: AUR, AUR3, Arch Subversion repositories, various
        Git repositories, local directories
    * search for info from registered sources
    * automatic resolving dependencies from registered sources
    * specify a tag to add to the version when building (e.g. python 3.3
        version vs python 3.2 version). Allows separate packages to be built
        and switched without conflicting the file or version name.
    * -u --sysupgrade
    * -r --rmdeps
    * -x --noinstall [aurbuild] -w --buildonly [pkgbuilder] (a bit like
        Pacman -w --downloadonly)
    * -o --builder-opts [aurbuild]; --mopt [aurget]: Passed to "makepkg"
    * -A --aursync [aura]: Like Pacman's -S --sync but for the AUR
    * Allow inspection and confirmation of sources before building, but
        minimise the repeditiveness of this when building many files
    * Allow a package to be deconfirmed
    * -w --downloadonly [aura; should be different to the binary download
        options]: Download source tarball into cache
    * AUR packages prefixed with aur/ repository name; incl. in search
        results
    * Model dependency building after Make; e.g. -k for keep building if
        possible after first failure
    * If a version number is specified; drop sources that do not provide that
        version
    * -n --noauto [aurora]: Build but do not install
    * -d --directory: Download directory [/tmp/aurora]
    * Clean list of packages build and where they came from, so you can go
        vote in AUR for them
    * Stick binary packages in Pacman cache? is there any point in this?
    * Should respect $PACMAN from makepkg.conf
    * Provide opportunity to edit source package even before its dependencies
        have been determined. But any modifications would be lost. Default
        should only be to view the source.
    * Simple mode to get source package and run makepkg without attempting to
        resolve dependencies itself, like Aurora
    * -U PKGBUILD or *.src.tar.gz equivalent to running "makepkg --install"
        [Packer, Meric]

makepkg and ABS
    * Platform: Bash, Pacman
    * Source files: /var/abs/
    * Packages pulled from rsync://rsync.archlinux.org/abs/$arch/$REPO/
        $pkgbase/
    * Allows individual repo or package to be specified to retrieve; probably
        ignored on first tarball retrieval though
    * Builds all packages by repo; user specified build directory
    * BUILDDIR=/tmp/makepkg PKGDEST=/home/packages SRCDEST=/home/sources
        SRCPKGDEST=/home/srcpackages
    * $BUILDDIR, $PKGDEST, $SRCDEST may be relative to current directory
    * $PKGDEST and $SRCDEST must already exist and be writable with original
        permissions; even if no downloads need to be stored
    * $BUILDDIR/src/; $BUILDDIR/pkg/ (originally). Now $BUILDDIR/$pkgname/
        {src,pkg}/
    * symbolically links ./*.pkg.* to $PKGDEST/*.pkg.*

Arson
    * Platform: Ruby
    * Presumed dead

Aura: Haskell; > 64 MiB dependencies

aurbuild: abandoned
    * Platform: Python 2 modules, Pacman, makepkg
    * Inspection: Menu before building each package
    * Permission: Must be invoked as root
    * Builds: /var/tmp/aurbuild/build/<pkgname>/<pkgname>/{PKGBUILD, src,
        pkg, $pkgname-$pkgver-$pkgrel-$arch$PKGEXT}; completely removed by
        default afterwards
    * Local source repository: /var/tmp/aurbuild/pkgbuilds/<pkgname>/PKGBUILD
        (sources optionally saved here)
    * UI: Custom Gnu CLI
    * Broken by AUR 2?

aurel: Emacs; does not build packages

aurget
    * Platform: Bash, Pacman, makepkg, Curl
    * Directories: ./$pkgbase/PKGBUILD, etc
    * Inspection: yes/no prompt
    * Runs Pacman without confirmations
    * Does not pay attention to makedeps; at least if a version number is
        specified

Aurifere (-git)
    * Platform: Python (3) modules, Git
    * Builds: ~/share/aurifere/<pkgname>/
    * Inspection: Git diffs

aurinstaller: disappeared

Aurora (-hg)
    * Platform: Single Python 3 script, Pacman, makepkg
    * Builds: /tmp/aurora/<pkgname>/
    * Permission: only via "makepkg"; option to drop from "root" to UID 99
        (nobody)
    * Does a substring search rather than taking a specific package name
    * Pauses but assumes separate terminal used to verify PKGBUILD file by
        default; --edit option as alternative
    * Uses "makepkg" to resolve dependencies, and only after editing source
        package! Nice for fixing up someone else's dependency list, but no
        automatic AUR dependencies.
    * Not Pacman UI compatible itself; options misleading (e.g. use --search
        to install something)

aursh (AUR shell)
    * Platform: Python 2
    * Presumed dead

aurpac: source included in AUR tarball

autoaur
    * Platform: Bash
    * Presumed dead
    * Has plugins for AUR, ABS, Pacman wrapping

Bauerbill
    * Platform: Perl (?)
    * Lots of nice ideas
    * No public copies found
    * https://wayback.archive.org/web/20100204133811/
        http://xyne.archlinux.ca/manpages/bauerbill

Clyde
    * Platform: Lua
    * Presumed dead

Cower
    * Platform: C, Curl, YAJL
    * Sources: ./<pkgname>/
    * Does not build packages

makeaur
    * Platform: Bash
    * Presumed dead

Meat (-git)
    * Platform: Bash, Cower, sudo, Pacman, makepkg
    * Builds: /tmp/tmp.*/$pkgname/PKGBUILD, etc
    * Inspection: Ask to edit PKGBUILD after confirming installation and
        after determining dependencies
    * Not Pacman compatible
    * CLI arguments cannot be reordered (Gnu style)

Meric
    * Fork of Packer
    * What was yellow on white is now black on red: marginally better

Owl (owlman):
    * Platform: Posix shell (uses Dash), sudo, Cower, Pacman
    * Looks like it completely wraps Pacman, adds lots of aliases, but with
        different CLI
    * Extremely brief CLI usage help
    * Install seems to rely on /home/src/aur/ (= XDG_AUR_HOME?) for AUR
        packages
    * Prints "running sudo" in white on white!

pacaur
    * Platform: Bash, expac, makepkg, Cower, sudo
    * Builds: /tmp/pacaurtmp-$USER/$pkgname
    * Some custom parser to handle dependencies based on $CARCH; maybe just
        using Bash after all :(
    * Prompt to edit PKGBUILD with VIM

pacmoon
    * ZSH
    * Usage similar to Portage
    * Handles dependencies, Arch repositories, recompiling precompiled Arch
        packages
    * Tracks packages that have been compiled; recognises when they are
        overwritten by a precompiled package
    * Possibly dead [untried]

Packer
    * Platform: Bash, $PACMAN, Curl, jshon, makepkg
    * Builds: /tmp/packerbuild-$UID/$pkgname/
        {$pkgname.tar.gz, $pkgname/PKGBUILD}
    * Inspection: edit each file (including install script) with $EDITOR
    * Permission: devious (automatically uses "sudo"?)
    * Pacman wrapper
    * No releases; expect whatever is at the head of repository
    * Horrible yellow on white colour output. Uses hard-coded strings like
        "-Ss".
    * Default operation is to do a search and install selected results from
        the search
    * Response to search-and-install must be a line, as for Pacman
        installation confirmation, but other confirmations are single
        character only
    * Does not show if AUR packages in search results are installed
    * pacman-mirrorlist [installed: newline]
    * Result of installing a package that doesn't exist looks more like a
        crash than an error message

paktahn: Common Lisp
pbfetch: Bash

pbget
    * Platform: Python 3 script (previously Bash), rsync, python3-aur, pyalpm
    * Does not build packages
    * Has various Arch repository architecture names hard-coded
    * Seems to scrape Git web pages with regular expressions!

pkgbuilder: Git repository history noisy and messed up
    * Platform: Python 3 modules, pyalpm, Pacman, makepkg, sudo
    * Builds: current directory, or /tmp/pkgbuilder-$UID/$pkgname/ if -S
        --sync mode ("pb" wrapper version only seems to do sync mode). Temp
        directory is kept after successful finish, with PKGBUILD, downloaded
        sources and build package file only (no src or pkg directory). Like
        makepkg --clean?
    * Source inspection: no chance!
    * <http://kwpolska.tk/blog/2012/09/19/pkgbuilder-ready-for-prime-time>
    * Get rid of Requests dependency if possible?
    
    Changes:
        * "Validating installation status... / NOT installed" should exit
            with failure status
        * Should allow source files to be inspected before executing them
        * Allow them to be edited as well, as a quick hack to fix errors,
            amend auto-detected dependencies, etc
        * Synchronise "repository" prefix: AUR's "aurifere-git" entry printed
            as "system/aurifere-git" but as printed it responds
        * Use PACMAN from makepkg.conf
        * Cancel "makepkg" dependency installation: pkgbuilder/build.py:123,
            in auto_build: TypeError: 'NoneType' object is not iterable
        * Seems to manually check for repository (Pacman installable)
            depencencies itself, but then use "makepkg" to actually install
            them. Perhaps install them ourselves?

pkgman
    Platform: Bash

powaur
    * Platform: C, Autocrap?, Curl, YAJL (JSON library), Pacman
    * Broken by AUR interface?
    * /tmp/powaur-$USER/

python3-aur: Library
pywer: Python 3; does not build packages
Slurpy: Python 2; presumed dead
Spinach: Bash
srcman: Bash; languishing: not sure where the functionality is

tusdah
    * Python
    * No dependency handling; Arch repository handling under development
    * Simple CLI: package name as argument
    * Maintains a repository built from the AUR
    * Not sure if being developed [untried]

trizen: Perl
wrapaur: Bash
yaah: Bash
Yaourt: Bash
Yogurt: dead

customizepkg

Source package building:
    curl "https://aur.archlinux.org/packages/${PKGNAME:0:2}/$PKGNAME/$PKGNAME.tar.gz" |
    tar xzvC ~/abs &&
    pushd ~/abs/"$PKGNAME" &&
    namcap -i PKGBUILD &&
    less PKGBUILD
    #
    makepkg &&
    pkg="$(. PKGBUILD && printf %s "$pkgname-$pkgver-$pkgrel"-*.pkg.tar.xz)" &&
    namcap "$pkg" &&
    sudo roopwn -U "$pkg" &&
    popd

Signed source packages:
    * include a PKGBUILD.sig in the source tarball
    * or maybe append signature to PKGBUILD, otherwise how will "makepkg"
        know to add it?
    * somehow someone needs to make sure that the signed PKGBUILD does not
        refer to extra files (eg install scripts) that are not mentioned and
        checksummed in the PKGBUILD file, otherwise it's too easy for someone
        to replace these files without invalidating the signature
    * Would be nice to easily be able to see who made a signature as well;
        just as an extra step; you still should verify if you like the person
        who signed it.
    * Maybe create tarball of source package, create signature file of it,
        append that to the tarball and upload that. Then it may be possible
        to verify the signature while you have the tar file, but not if you
        delete the tar file.
    * Sign all other files added to tar file in the one signature file. Then
        you can verify these files after extracting and removing the tar
        file, but you still have to be careful of referencing unsigned files
        from the PKGBUILD, so maybe not much better than just signing the
        PKGBUILD.
    * Would like to avoid someone being able to include unsigned files in a
        tarball. Obviously easy to prevent removal, modification, and
        renaming of signed files. Possibly also sign other aspects of the
        files, such as permissions (raises portability issue), names of empty
        directories. Perhaps even signing timestamps, but that would be even
        less portable.
"""

from sys import argv
import subprocess
from sys import stderr
from posixpath import basename
import os
import tarfile
from os import (fsdecode, fsencode)
from stat import (S_IXUSR, S_IXGRP, S_IXOTH)
from os.path import splitext
from types import SimpleNamespace
from shutil import copyfileobj
from gzip import GzipFile
from bz2 import BZ2File
from tempfile import TemporaryDirectory
from deps import (Deps, Filesystem, OsFilesystem, is_privileged, LibCache)
from functools import partial
from os.path import exists
from errno import (EINVAL, ENOENT, ENOTDIR)
from os import (strerror, stat)
from signal import (signal, SIGINT, SIGQUIT, SIG_IGN)
from collections import defaultdict
from os import devnull
from operator import itemgetter
import elf
from elftools.elf.elffile import ELFFile
from contextlib import (ExitStack, closing)
from os import execvp
from lzma import LZMAFile
from elftools.common.exceptions import ELFError
import posixpath

CACHEDIR = b"/var/cache/pacman/pkg"

def main():
    through = list()
    sync = []
    upgrade = []
    remove = []
    refresh = []
    xaction_args = list()
    dryrun = False
    filedeps = True
    verbose = False
    help = False
    pacman = "pacman"
    allsymbols = False
    symboldeps = True
    
    args = iter(argv[1:])
    for arg in args:
        while True:
            strung = arg.startswith("-") and arg[1] != "-" and len(arg) > 2
            if strung:
                opt = arg[:1 + 1]
            else:
                opt = arg
            
            if opt in ("-S", "--sync"):
                sync.append(opt)
            elif opt in ("-U", "--upgrade"):
                upgrade.append(opt)
            elif opt in ("-R", "--remove"):
                remove.append(opt)
            elif opt in ("-y", "--refresh"):
                refresh.append(opt)
            elif opt in (
                "--needed", "--asdeps", "--asexplicit", "--noscriptlet",
                "-d", "--nodeps", "--noprogressbar", "-f", "--force",
                "--recursive", "--noconfirm", "-c", "--cascade",
                "-n", "--nosave", "-s", "-u", "--unneeded",
            ):
                xaction_args.append(opt)
            elif opt in ("--dryrun", "--dry-run"):
                dryrun = True
            elif opt == "--nofiledeps":
                filedeps = False
            elif opt == "--allsymbols":
                allsymbols = True
            elif opt == "--nosymboldeps":
                symboldeps = False
            elif opt in ("-v", "--verbose"):
                verbose = True
            elif opt in ("-h", "--help"):
                help = True
            elif opt == "--pacman":
                pacman = next(args)
            else:
                through.append(arg)
                if opt.startswith("-"):
                    through.extend(args)
                    break
            
            if strung:
                arg = "-" + arg[1 + 1:]
            else:
                break
    
    if help:
        print("""\
Parameters: <operation> [options . . .] [Pacman arguments . . .]
Operations handled natively: -S --sync -U --upgrade -R --remove
Other operations are passed to Pacman.
Options handled natively:
    --nofiledeps --allsymbols --nosymboldeps
    --dryrun
    -v --verbose
    -h --help
    --pacman: Specify command to run for Pacman
Options passed to Pacman:
    -y --refresh
    --needed -s --recursive -u --unneeded
    -c --cascade
    --asdeps --asexplicit
    --noscriptlet -d --nodeps -f --force
    --noprogressbar --noconfirm
    -n --nosave""",
        file=stderr)
        return
    
    if not sync and not upgrade and not remove:
        proc = [pacman] + refresh + xaction_args + through
        if verbose:
            command_trace(proc)
        execvp(pacman, proc)
    
    if upgrade:
        pkg_files = through
    
    if sync:
        proc = [pacman] + sync + refresh + xaction_args
        command(proc + ["--downloadonly"] + through, trace=verbose)
        
        pkg_files = list()
        proc = [pacman] + sync + xaction_args + ["--print"] + through
        with command_pipe(proc, stdout=subprocess.PIPE,
        trace=verbose) as proc:
            for pkg in proc.stdout:
                (pkg,) = pkg.splitlines()
                pkg_files.append(os.path.join(CACHEDIR, basename(pkg)))
        
        if not pkg_files:
            return
    
    if remove:
        proc = ([pacman] + remove + refresh + xaction_args + ["--print"] +
            through)
        with command_pipe(proc, stdout=subprocess.PIPE, check=False,
        trace=verbose) as proc:
            stdout = proc.stdout.read()
        
        # Pacman prints messages to stdout, as well as package names!
        if proc.returncode or b" " in stdout:
            stderr.buffer.write(stdout)
            raise SystemExit(proc.returncode)
        
        pkg_names = list()
        for pkg in stdout.splitlines():
            (pkg, _) = pkg_parse(pkg)
            pkg_names.append(pkg)
    
    anal = Analyser(pacman, allsymbols, symboldeps, verbose=verbose)
    with ExitStack() as cleanup:
        print("analysing file dependencies...", file=stderr)
        
        if not remove:
            work = cleanup.enter_context(TemporaryDirectory(prefix="roopwn"))
            
            if verbose:
                print("extracting packages...", file=stderr)
            pkg_names = list()
            for pkg in pkg_files:
                tar = tarfile.open(decompress(pkg, ".tar", work))
                cleanup.enter_context(tar)
                
                PKG = ".pkg."
                DASH = "-"
                if isinstance(pkg, bytes):
                    PKG = fsencode(PKG)
                    DASH = fsencode(DASH)
                pkg = basename(pkg).rsplit(PKG, 1)[-2].rsplit(DASH, 1)[0]
                (name, _) = pkg_parse(pkg)
                pkg_names.append(name)
                
                if not anal.fs.add(pkg, tar):
                    anal.fail = True
        
        # Get list of files which may be removed
        recheck = set()
        realdir = partial(PackagesFs.realdir, OsFilesystem())
        proc = [pacman] + "--query --list --quiet --".split() + pkg_names
        with command_pipe(proc, stdout=subprocess.PIPE,
        stderr=command_pipe.NULL, check=False, trace=verbose) as proc:
            for file in proc.stdout:
                (file,) = file.splitlines()
                file = realdir(strip_root(file))
                anal.fs.removed.add(file)
                recheck.update(anal.deps.pop(file, ()))
        
        if not remove:
            if verbose:
                print("searching for new dependencies...", file=stderr)
            
            if filedeps:
                def on_needed(deps, needed):
                    (dir, so) = os.path.split(needed["name"])
                    try:
                        so = so.rsplit(b".so.", 1)[-2]
                    except IndexError:
                        return
                    so = os.path.join(dir, so + b".so")
                    if needed["search"]:
                        for match in deps.search_lib(so, anal.cache):
                            anal.fs.owner(match)
                    else:
                        if anal.fs.exists(so):
                            anal.fs.owner(so)
            else:
                on_needed = None
            
            for (filename, entry) in anal.fs.paths.items():
                recheck.discard(filename)
                
                member = entry["member"]
                if not member.isfile():
                    continue
                
                file = entry["tar"].extractfile(member)
                anal.analyse(file, filename, member.mode,
                    on_needed=on_needed)
        
        if verbose:
            print("updating old dependencies...", file=stderr)
        while recheck:
            dep = recheck.pop()
            try:
                file = anal.fs.open(dep)
            except EnvironmentError:
                continue
            with file:
                fail = not anal.analyse(file, dep, anal.fs.stat(dep).st_mode)
            if fail and filedeps:
                anal.fs.owner(dep)
    
    if filedeps and anal.fail:
        raise SystemExit(1)
    
    if not dryrun:
        if remove:
            proc = [pacman] + remove + xaction_args + ["--"] + pkg_names
            command(proc, trace=verbose)
        else:
            proc = [pacman, "--upgrade"]
            if sync:
                proc.append("--noconfirm")
            command(proc + xaction_args + ["--"] + pkg_files, trace=verbose)
        anal.write_db()

class Analyser(object):
    def __init__(self, pacman, allsymbols, symboldeps, verbose):
        self.allsymbols = allsymbols
        self.symboldeps = symboldeps
        self.verbose = verbose
        
        # A set of dependent ("client") files for each independent
        # ("supplier") file. Some dependent files may no longer actually
        # exist or be dependent, because they are not delisted when their
        # package is touched.
        self.deps = defaultdict(set)
        
        self.broken = defaultdict(set)  # Broken dependencies to ignore
        self.read_db()
        
        self.fs = PackagesFs(pacman)
        self.fail = False
        self.cache = LibCache(self.fs)
    
    def analyse(self, file, filename, mode, on_needed=None):
        fail = False
        
        origin = partial(self.fs.get_origin, filename)
        privileged = is_privileged(mode)
        try:
            file = ELFFile(file)
            segments = elf.Segments(file)
            if not segments:  # Not a dynamically linked file
                return
            dynamic = segments.read_dynamic()
            deps = Deps(file, origin, privileged,
                segments=segments, dynamic=dynamic)
        except (ELFError, LookupError):
            return
        
        known_broken = self.broken[filename]
        now_broken = list()
        
        for interp in deps.interps():
            if self.fs.exists(interp):
                interp = self.fs.realdir(interp)
                self.deps[interp].add(filename)
                if self.verbose:
                    print("{}: interp -> {}".format(
                        fsdecode(filename), fsdecode(interp)), file=stderr)
            else:
                dep = (b"interp", interp)
                now_broken.append(dep)
                unknown = dep not in known_broken
                if unknown or self.vebose:
                    print("{}: {}: interp not found".format(
                        fsdecode(filename), fsdecode(interp)), file=stderr)
                fail |= unknown
        
        needed_paths = list()
        needed_missing = False
        for needed in deps.needed():
            if needed["search"]:
                found = deps.search_lib(needed["name"], self.cache)
                found = next(found, None)
            else:
                found = needed["name"]
                if not self.fs.exists(found):
                    found = None
            
            if found is None:
                dep = (b"needed", needed["raw_name"])
                now_broken.append(dep)
                unknown = dep not in known_broken
                if unknown or self.verbose:
                    print("{}: {}: needed not found".format(
                        fsdecode(filename), fsdecode(needed["raw_name"])),
                        file=stderr)
                if unknown:
                    if on_needed:
                        on_needed(deps, needed)
                    fail = True
                needed_missing = True
                continue
            
            found = self.fs.realdir(found)
            self.deps[found].add(filename)
            needed_paths.append(found)
            if self.verbose:
                if found == needed["raw_name"]:
                    print("{}: needed -> {}".format(
                        fsdecode(filename), fsdecode(found)), file=stderr)
                else:
                    print("{}: needed {} -> {}".format(fsdecode(filename),
                        fsdecode(needed["raw_name"]), fsdecode(found)),
                        file=stderr)
        
        known_broken.clear()
        known_broken.update(now_broken)
        
        if needed_missing or not self.symboldeps:
            self.fail |= fail
            return not fail
        if self.allsymbols:
            if self.verbose:
                print("{}: checking symbols".format(fsdecode(filename)),
                    file=stderr)
        else:
            if self.verbose:
                print("{}: {}".format(fsdecode(filename), file["e_type"]),
                    file=stderr)
            if file["e_type"] != "ET_EXEC":
                return not fail
        
        with ExitStack() as context:
            symtab = dynamic.symbol_table()
            
            hashes = list()
            hashes.append(dynamic.symbol_hash(symtab))
            for file in needed_paths:
                file = context.enter_context(closing(self.fs.open(file)))
                ddynamic = elf.Segments(ELFFile(file)).read_dynamic()
                dsymtab = ddynamic.symbol_table()
                hashes.append(ddynamic.symbol_hash(dsymtab))
            
            # Find unresolved symbol references in the file being analysed
            for rel in dynamic.rel_entries():
                if not rel['r_info_sym']:
                    continue
                sym = symtab[rel['r_info_sym']]
                
                # Ignore locally bound symbols because their resolution is
                # not affected by external dependencies. Ignore weak symbols,
                # since it looks like they default to zero if not defined.
                if sym['st_info']['bind'] in ('STB_LOCAL', 'STB_WEAK'):
                    continue
                
                # STV_DEFAULT is zero
                if sym['st_other']['visibility'] != 'STV_DEFAULT':
                    continue
                
                for hash in hashes:
                    try:
                        defin = hash[sym.name]
                    except LookupError:
                        continue
                    if (defin['st_info']['bind'] != 'STB_LOCAL' and
                    defin['st_other']['visibility'] not in (
                    'STV_HIDDEN', 'STV_INTERNAL')):
                        break
                else:
                    print("{}: unresolved symbol: {}".format(
                        fsdecode(filename), fsdecode(sym.name)),
                        file=stderr)
                    fail |= self.symboldeps
        
        self.fail |= fail
        return not fail
    
    def read_db(self):
        filename = os.path.join(self.DB_DIR, self.DEP_DB)
        try:
            db = open(filename, "rb")
        except FileNotFoundError as err:
            print(err, file=stderr)
        else:
            with db:
                for line in db:
                    line = line.rstrip(b"\n")
                    if line.startswith(b"  "):
                        deps.add(line[2:])
                    else:
                        indep = line.rstrip(b":")
                        deps = self.deps[indep]
        
        filename = os.path.join(self.DB_DIR, self.BROKEN_DB)
        try:
            db = open(os.path.join(self.DB_DIR, self.BROKEN_DB), "rb")
        except FileNotFoundError as err:
            print(err, file=stderr)
        else:
            with db:
                for line in db:
                    line = line.rstrip(b"\n")
                    if line.startswith(b"  "):
                        broken.add(tuple(line[2:].split(b":")))
                    else:
                        broken = self.broken[line.rstrip(b":")]
    
    def write_db(self):
        os.makedirs(self.DB_DIR, exist_ok=True)
        
        with open(os.path.join(self.DB_DIR, self.DEP_DB), "wb") as db:
            for (indep, deps) in sorted(self.deps.items()):
                db.writelines((indep, b":"))
                for dep in deps:
                    db.writelines((b"\n  ", dep))
                db.write(b"\n")
        
        with open(os.path.join(self.DB_DIR, self.BROKEN_DB), "wb") as db:
            for (filename, broken) in self.broken.items():
                if not broken:
                    continue
                db.writelines((filename, b":"))
                for dep in broken:
                    db.writelines((b"\n  ", b":".join(dep)))
                db.write(b"\n")
    
    DB_DIR = "/var/lib/roopwn"
    DEP_DB = "deps"
    BROKEN_DB = "broken"

class PackagesFs(Filesystem):
    def __init__(self, pacman):
        self.pacman = pacman
        self.removed = set()
        self.paths = dict()
    
    def add(self, pkg, tar):
        success = True
        for member in tar:
            filename = strip_root(fsencode(member.name))
            if filename.startswith(b"."):
                continue
            filename = self.realdir(filename)
            
            if filename in self.paths:
                if member.isdir():
                    continue
                
                (old, _) = pkg_parse(self.paths[filename]["pkg"])
                (new, _) = pkg_parse(pkg)
                print(fsdecode(new), "duplicates", member.name, "from",
                    fsdecode(old), file=stderr)
                success = False
            
            self.paths[filename] = dict(pkg=pkg, tar=tar, member=member)
        
        return success
    
    def realdir(self, path):
        (dir, base) = posixpath.split(path)
        return posixpath.join(self.realpath(dir), base)
    
    def lookup(self, path):
        file = self.paths.get(path)
        if not file and path in self.removed:
            raise OSError(ENOENT, "File removed", path)
        return file
    
    # TODO: expand links in parent directories before looking up tar file members
    
    def open(self, path):
        file = self.lookup(self.realpath(path))
        if not file:
            return OsFilesystem.open(self, path)
        
        file = file["tar"].extractfile(file["member"])
        if not file:
            raise ErrnoError(EINVAL, path)
        return file
    
    def exists(self, path):
        realpath = self.realpath(path)
        if realpath in self.paths:
            return True
        if realpath in self.removed:
            return False
        return exists(b"/" + path)
    
    def readlink(self, path):
        # Not bothering to call realpath() on the parent directory because
        # this is only called by realpath() itself which has already expanded
        # the directory
        file = self.lookup(path)
        if not file:
            return OsFilesystem.readlink(self, path)
        
        member = file["member"]
        if not member.issym():
            raise ErrnoError(EINVAL, path)
        return fsencode(member.linkname)
    
    def listdir(self, path):
        realpath = self.realpath(path)
        pkg_dir = self.lookup(realpath)
        if pkg_dir and not pkg_dir["member"].isdir():
            raise ErrnoError(ENOTDIR, path)
        
        res = list()
        prefix = realpath + b"/"
        for entry in self.paths.keys():
            if not entry.startswith(prefix):
                continue
            entry = entry[len(prefix):]
            
            if b"/" not in entry:
                res.append(entry)
        
        try:
            res.extend(OsFilesystem.listdir(self, path))
        except EnvironmentError as err:
            if err.errno != ENOENT or not pkg_dir:
                raise
        
        return res
    
    def owner(self, path):
        file = self.lookup(self.realpath(path))
        if not file:
            stderr.write(":: ")
            stderr.flush()
            command([self.pacman] + "--query --owns --".split() +
                [b"/" + path])
        else:
            (pkg, ver) = pkg_parse(file["pkg"])
            print(":: {} would be installed by {} {}".format(
                fsdecode(path), fsdecode(pkg), fsdecode(ver)), file=stderr)
    
    def stat(self, path):
        file = self.lookup(self.realpath(path))
        if not file:
            return stat(b"/" + path)
        else:
            return SimpleNamespace(st_mode=file["member"].mode)

def pkg_parse(name):
    DASH = "-"
    if isinstance(name, bytes):
        DASH = fsencode(DASH)
    (name, ver, rel) = name.rsplit(DASH, 2)
    return (name, DASH.join((ver, rel)))

def decompress(file, ext, dir):
    (decomp_file, comp_ext) = splitext(fsdecode(file))
    
    if not comp_ext:
        return file
    comp_ext = comp_ext[1:]  # Strip full stop prefix
    
    try:
        type = comp_types[comp_ext]
    except LookupError:
        return file
    
    decomp_file = basename(decomp_file)
    try:
        ext = type["ext"]
    except LookupError:
        (_, ext2) = splitext(decomp_file)
        if ext2 == ext:
            ext = ""
    else:
        ext = "." + ext
    
    decomp_file = os.path.join(dir, decomp_file + ext)
    
    with type["comp"](file) as src:
        with open(decomp_file, "wb") as dest:
            copyfileobj(src, dest)
    
    return decomp_file

comp_types = dict(
    gz=dict(comp=GzipFile), tgz=dict(comp=GzipFile, ext="tar"),
    bz2=dict(comp=BZ2File), tbz=dict(comp=BZ2File, ext="tar"),
    lzma=dict(comp=LZMAFile),
    xz=dict(comp=LZMAFile), txz=dict(comp=LZMAFile, ext="tar"),
)

def command(*pos, **kw):
    """
    Similar to Posix's system() function. It should ignore interrupt and quit
    signals in the parent while waiting. These signals are normally sent to
    both the parent and child.
    """
    
    # There is a small window where the signals will be completely
    # lost, before forking the child. But there doesn't seem to be
    # a way to avoid it. Gnu C library:
    # http://sourceware.org/git?p=glibc.git;a=blob;f=sysdeps/posix/system.c
    with IgnoreSigs() as sigs, \
            command_pipe(*pos, check=True, preexec_fn=sigs.__exit__, **kw):
        pass

class IgnoreSigs(object):
    def __enter__(self):
        self.int = None
        self.quit = None
        try:
            self.int = signal(SIGINT, SIG_IGN)
            self.quit = signal(SIGQUIT, SIG_IGN)
            return self
        except:
            self.__exit__()
            raise
    
    def __exit__(self, *args, **kw):
        if self.int is not None:
            signal(SIGINT, self.int)
        if self.quit is not None:
            signal(SIGQUIT, self.quit)

class command_pipe(subprocess.Popen):
    def __init__(self,
    proc_args, *popen_args,
    bufsize=-1, stderr=None, trace=False, check=True, **kw):
        self.trace = trace
        self.check = check
        
        if stderr is self.NULL:
            null = open(devnull, "w")
        else:
            null = None
        
        try:
            if stderr is self.NULL:
                stderr = null
            
            if self.trace:
                self.arg0 = proc_args[0]
                command_trace(proc_args)
            
            subprocess.Popen.__init__(self,
                proc_args, *popen_args,
                bufsize=bufsize, stderr=stderr, **kw)
        
        finally:
            if null:
                null.close()
    
    def wait(self, *args, **kw):
        returncode = subprocess.Popen.wait(self, *args, **kw)
        if self.check and returncode:
            if self.trace:
                print("Command", repr(self.arg0),
                    "failed with exit status", returncode, file=stderr)
            raise SystemExit(returncode)
        return returncode
    
    NULL = object()

def command_trace(args):
    print("+", " ".join(fsdecode(arg) for arg in args), file=stderr)

def ErrnoError(errno, *args, **kw):
    return EnvironmentError(errno, strerror(errno), *args, **kw)

def strip_root(path):
    start = 0
    while True:
        for prefix in (b"/", b"./", b"../"):
            if path.startswith(prefix, start):
                start += len(prefix)
                break
        else:
            return path[start:]

if __name__ == "__main__":
    main()