Skip to content
Browse files

Merge branch 'devel'

  • Loading branch information...
2 parents 71951c9 + 96d0010 commit 3e33bb1e06232605565f5deb323fd6fd783f604f @dvarrazzo committed Sep 29, 2012
View
27 AUTHORS
@@ -0,0 +1,27 @@
+Who has contributed to the PGXN client?
+=======================================
+
+Daniele Varrazzo
+
+ He rushed to implement a client before David could do it in Perl!
+
+David Wheeler
+
+ He is the PGXN mastermind: a lot of helpful design discussions.
+
+Peter Eisentraut
+
+ First implementation of tarball support. Auto-sudo is not a good idea, I
+ got it.
+
+Hitoshi Harada
+
+ Tricky installation corner cases.
+
+Andrey Popp
+
+ Make selection. Helped the program not to suck on BSD!
+
+Also thank you everybody for the useful discussions on the PGXN mailing list,
+bug reports, proofreading the docs and the general support to the project.
+
View
11 CHANGES
@@ -3,6 +3,17 @@
PGXN Client changes log
-----------------------
+pgxnclient 1.2
+==============
+
+- Packages can be downloaded, installed, loaded specifying an URL
+ (ticket #15).
+- Added support for ``.tar`` files (ticket #17).
+- Use ``gmake`` in favour of ``make`` for platforms where the two are
+ distinct, such as BSD (ticket #14).
+- Added ``--make`` option to select the make executable (ticket #16).
+
+
pgxnclient 1.1
==============
View
2 MANIFEST.in
@@ -1,4 +1,4 @@
-include CHANGES COPYING MANIFEST.in README.rst setup.py Makefile
+include AUTHORS CHANGES COPYING MANIFEST.in README.rst setup.py Makefile
include bin/pgxn bin/pgxnclient
recursive-include pgxnclient *.py
recursive-include testdata *
View
6 docs/index.rst
@@ -20,16 +20,16 @@ which would load the extension in one of the databases of the server.
The client interacts with the PGXN web service and a ``Makefile`` provided by
the extension. The best results are achieved with makefiles using the
PostgreSQL `Extension Building Infrastructure`__; however the client tries to
-degrade gracefully in presence of any package hosted on PGXN.
+degrade gracefully in presence of any package hosted on PGXN and any package
+available outside the extension network.
.. _semver: http://pgxn.org/dist/semver
-.. __: http://www.postgresql.org/docs/9.1/static/extend-pgxs.html
+.. __: http://www.postgresql.org/docs/current/static/extend-pgxs.html
- Home page: http://pgxnclient.projects.postgresql.org/
- Downloads: http://pypi.python.org/pypi/pgxnclient/
- Discussion group: http://groups.google.com/group/pgxn-users/
- Source repository: https://github.com/dvarrazzo/pgxnclient/
-- PgFoundry project: http://pgfoundry.org/projects/pgxnclient/
Contents:
View
80 docs/usage.rst
@@ -60,10 +60,17 @@ Whenever a command takes a specification in input, it also accepts options
``--stable``, ``--testing`` and ``--unstable`` to specify the minimum release
status accepted. The default is "stable".
-A few commands also allow specifying a local ``.zip`` package or a local
-directory containing a distribution: in this case the specification should
-contain at least a path separator to disambiguate it from a distribution name,
-for instance ``pgxn install ./foo.zip``.
+A few commands also allow specifying a local archive or local directory
+containing a distribution: in this case the specification should contain at
+least a path separator to disambiguate it from a distribution name (for
+instance ``pgxn install ./foo.zip``) or it should be specified as an URL with
+``file://`` schema.
+
+A few commands also allow specifying a remote package with a URL. Currently
+the schemas ``http://`` and ``https://`` are supported.
+
+Currently the client supports ``.zip`` and ``.tar`` archives (eventually with
+*gzip* and *bz2* compression).
.. _install:
@@ -79,12 +86,14 @@ Usage:
:class: pgxn-install
pgxn install [--help] [--stable | --testing | --unstable]
- [--pg_config *PATH*] [--sudo [*PROG*] | --nosudo]
+ [--pg_config *PROG*] [--make *PROG*]
+ [--sudo [*PROG*] | --nosudo]
*SPEC*
The program takes a `package specification`_ identifying the distribution to
work with. The download phase is skipped if the distribution specification
-refers to a local directory or package.
+refers to a local directory or package. The package may be specified with an
+URL.
Note that the built extension is not loaded in any database: use the command
`load`_ for this purpose.
@@ -95,13 +104,20 @@ then will perform ``make all`` and ``make install``. It is assumed that the
but this is not enforced: you may provide any Makefile as long as the expected
commands are implemented.
-.. _PGXS: http://www.postgresql.org/docs/9.1/static/extend-pgxs.html
+.. _PGXS: http://www.postgresql.org/docs/current/static/extend-pgxs.html
If there are many PostgreSQL installations on the system, the extension will
be built and installed against the instance whose :program:`pg_config` is
first found on the :envvar:`PATH`. A different instance can be specified using
the option :samp:`--pg_config {PATH}`.
+The PGXS_ build system relies on a presence of `GNU Make`__: in many systems
+it is installed as :program:`gmake` or :program:`make` executable. The program
+will use the first of them on the path. You can specify an alternative program
+using ``--make`` option.
+
+.. __: http://www.gnu.org/software/make/
+
If the extension is being installed into a system PostgreSQL installation, the
install phase will likely require root privileges to be performed. In this
case either run the command under :program:`sudo` or specify the ``--sudo``
@@ -131,12 +147,14 @@ Usage:
:class: pgxn-check
pgxn check [--help] [--stable | --testing | --unstable]
- [--pg_config *PATH*] [-d *DBNAME*] [-h *HOST*] [-p *PORT*] [-U *NAME*]
+ [--pg_config *PROG*] [--make *PROG*]
+ [-d *DBNAME*] [-h *HOST*] [-p *PORT*] [-U *NAME*]
*SPEC*
The command takes a `package specification`_ identifying the distribution to
-work with, which can also be a local file or directory. The distribution is
-unpacked if required and the ``installcheck`` make target is run.
+work with, which can also be a local file or directory or an URL. The
+distribution is unpacked if required and the ``installcheck`` make target is
+run.
.. note::
The command doesn't run ``make all`` before ``installcheck``: if any file
@@ -159,6 +177,8 @@ The database connection options are similar to the ones in load_, with the
difference that the variable :envvar:`PGDATABASE` doesn't influence the
database name.
+See the install_ command for details about the command arguments.
+
.. warning::
At the time of writing, :program:`pg_regress` on Debian and derivatives is
affected by `bug #554166`__ which makes *HOST* selection impossible.
@@ -179,7 +199,8 @@ Usage:
:class: pgxn-uninstall
pgxn uninstall [--help] [--stable | --testing | --unstable]
- [--pg_config *PATH*] [--sudo [*PROG*] | --nosudo]
+ [--pg_config *PROG*] [--make *PROG*]
+ [--sudo [*PROG*] | --nosudo]
*SPEC*
The command does the opposite of the install_ command, removing a
@@ -212,11 +233,11 @@ Usage:
*SPEC* [*EXT* [*EXT* ...]]
The distribution is specified according to the `package specification`_ and
-can refer to a local directory or file. No consistency check is performed
-between the packages specified in the ``install`` and ``load`` command: the
-specifications should refer to compatible packages. The specified distribution
-is only used to read the metadata: only installed files are actually used to
-issue database commands.
+can refer to a local directory or file or to an URL. No consistency check is
+performed between the packages specified in the ``install`` and ``load``
+command: the specifications should refer to compatible packages. The specified
+distribution is only used to read the metadata: only installed files are
+actually used to issue database commands.
The database to install into can be specified using options
``-d``/``--dbname``, ``-h``/``--host``, ``-p``/``--port``,
@@ -235,8 +256,8 @@ extension specifies a ``.control`` file, it will be loaded using the `CREATE
EXTENSION`_ command, otherwise it will be loaded as a loose set of objects.
For more information see the `extensions documentation`__.
-.. _CREATE EXTENSION: http://www.postgresql.org/docs/9.1/static/sql-createextension.html
-.. __: http://www.postgresql.org/docs/9.1/static/extend-extensions.html
+.. _CREATE EXTENSION: http://www.postgresql.org/docs/current/static/sql-createextension.html
+.. __: http://www.postgresql.org/docs/current/static/extend-extensions.html
The command is based on the `'provides' section`_ of the distribution's
``META.json``: if a SQL file is specified, that file will be used to load the
@@ -249,7 +270,7 @@ confirmation.
If the distribution provides more than one extension, the extensions are
loaded in the order in which they are specified in the ``provides`` section of
-the ``META.json`` file. It is also possilbe to load only a few of the
+the ``META.json`` file. It is also possible to load only a few of the
extensions provided, specifying them after *SPEC*: the extensions will be
loaded in the order specified.
@@ -297,11 +318,11 @@ itself, so the option will be ignored.
If the distribution specifies more than one extension, they are unloaded in
reverse order respect to the order in which they are specified in the
-``META.json`` file. It is also possilbe to unload only a few of the
+``META.json`` file. It is also possible to unload only a few of the
extensions provided, specifying them after *SPEC*: the extensions will be
unloaded in the order specified.
-.. _DROP EXTENSION: http://www.postgresql.org/docs/9.1/static/sql-dropextension.html
+.. _DROP EXTENSION: http://www.postgresql.org/docs/current/static/sql-dropextension.html
See the load_ command for details about the command arguments.
@@ -322,11 +343,12 @@ Usage:
[--target *PATH*]
*SPEC*
-The distribution is specified according to the `package specification`_. The
-file is saved in the current directory with name usually
-:samp:`{distribution}-{version}.zip`. If a file with the same name exists, a
-suffix ``-1``, ``-2`` etc. is added to the name, before the extension. A
-different directory or name can be specified using the ``--target`` option.
+The distribution is specified according to the `package specification`_ and
+can be represented by an URL. The file is saved in the current directory with
+name usually :samp:`{distribution}-{version}.zip`. If a file with the same
+name exists, a suffix ``-1``, ``-2`` etc. is added to the name, before the
+extension. A different directory or name can be specified using the
+``--target`` option.
.. _pgxn-search:
@@ -407,9 +429,9 @@ Usage:
[--details | --meta | --readme | --versions]
*SPEC*
-The distribution is specified according to the `package specification`_.
-The command output is a list of values obtained by the distribution's
-``META.json`` file, for example:
+The distribution is specified according to the `package specification`_. It
+cannot be a local dir or file nor an URL. The command output is a list of
+values obtained by the distribution's ``META.json`` file, for example:
.. code-block:: console
View
2 pgxnclient/__init__.py
@@ -6,7 +6,7 @@
# This file is part of the PGXN client
-__version__ = '1.1'
+__version__ = '1.2'
# Paths where to find the command executables.
# If relative, it's from the `pgxnclient` package directory.
View
6 pgxnclient/api.py
@@ -10,9 +10,9 @@
from urllib import urlencode
+from pgxnclient import network
from pgxnclient.utils import load_json
from pgxnclient.errors import NetworkError, NotFound, ResourceNotFound
-from pgxnclient.network import get_file
from pgxnclient.utils.uri import expand_template
@@ -78,7 +78,7 @@ def user(self, username):
def call(self, meth, args=None, query=None):
url = self.get_url(meth, args, query)
- return get_file(url)
+ return network.get_file(url)
def get_url(self, meth, args=None, query=None):
tmpl = self.get_template(meth)
@@ -98,7 +98,7 @@ def get_index(self):
if self._api_index is None:
url = self.mirror.rstrip('/') + '/index.json'
try:
- with get_file(url) as f:
+ with network.get_file(url) as f:
self._api_index = load_json(f)
except ResourceNotFound:
raise NetworkError("API index not found at '%s'" % url)
View
99 pgxnclient/archive.py
@@ -0,0 +1,99 @@
+"""
+pgxnclient -- archives handling
+"""
+
+# Copyright (C) 2011-2012 Daniele Varrazzo
+
+# This file is part of the PGXN client
+
+import os
+
+from pgxnclient.i18n import _
+from pgxnclient.utils import load_jsons
+from pgxnclient.errors import PgxnClientException
+
+def from_spec(spec):
+ """Return an `Archive` instance to handle the file requested by *spec*
+ """
+ assert spec.is_file()
+ return from_file(spec.filename)
+
+def from_file(filename):
+ """Return an `Archive` instance to handle the file *filename*
+ """
+ from pgxnclient.zip import ZipArchive
+ from pgxnclient.tar import TarArchive
+
+ for cls in (ZipArchive, TarArchive):
+ a = cls(filename)
+ if a.can_open():
+ return a
+
+ raise PgxnClientException(
+ _("can't open archive '%s': file type not recognized")
+ % filename)
+
+
+class Archive(object):
+ """Base class to handle archives."""
+ def __init__(self, filename):
+ self.filename = filename
+
+ def can_open(self):
+ """Return `!True` if the `!filename` can be opened by the obect."""
+ raise NotImplementedError
+
+ def open(self):
+ """Open the archive for usage.
+
+ Raise PgxnClientException if the archive can't be open.
+ """
+ raise NotImplementedError
+
+ def close(self):
+ """Close the archive after usage."""
+ raise NotImplementedError
+
+ def list_files(self):
+ """Return an iterable with the list of file names in the archive."""
+ raise NotImplementedError
+
+ def read(self, fn):
+ """Return a file's data from the archive."""
+ raise NotImplementedError
+
+ def unpack(self, destdir):
+ raise NotImplementedError
+
+ def get_meta(self):
+ filename = self.filename
+
+ self.open()
+ try:
+ # Return the first file with the expected name
+ for fn in self.list_files():
+ if fn.endswith('META.json'):
+ return load_jsons(self.read(fn).decode('utf8'))
+ else:
+ raise PgxnClientException(
+ _("file 'META.json' not found in archive '%s'") % filename)
+ finally:
+ self.close()
+
+ def _find_work_directory(self, destdir):
+ """
+ Choose the directory where to work.
+
+ Because we are mostly a wrapper for pgxs, let's look for a makefile.
+ The tar should contain a single base directory, so return the first
+ dir we found containing a Makefile, alternatively just return the
+ unpacked dir
+ """
+ for dir in os.listdir(destdir):
+ for fn in ('Makefile', 'makefile', 'GNUmakefile', 'configure'):
+ if os.path.exists(os.path.join(destdir, dir, fn)):
+ return os.path.join(destdir, dir)
+
+ return destdir
+
+
View
3 pgxnclient/cli.py
@@ -122,3 +122,6 @@ def _get_exec(cmd):
return fn
+if __name__ == '__main__':
+ script()
+
View
146 pgxnclient/commands/__init__.py
@@ -17,14 +17,16 @@
import logging
from subprocess import Popen, PIPE
-from pgxnclient.utils import load_json
-from pgxnclient.utils import argparse
+from pgxnclient.utils import load_json, argparse, find_executable
from pgxnclient import __version__
+from pgxnclient import network
from pgxnclient import Spec, SemVer
+from pgxnclient import archive
from pgxnclient.api import Api
from pgxnclient.i18n import _, gettext
from pgxnclient.errors import NotFound, PgxnClientException, ProcessError, ResourceNotFound, UserAbort
+from pgxnclient.utils.temp import temp_dir
logger = logging.getLogger('pgxnclient.commands')
@@ -224,7 +226,6 @@ def popen(self, cmd, *args, **kwargs):
from pgxnclient.errors import BadSpecError
-from pgxnclient.utils.zip import get_meta_from_zip
class WithSpec(Command):
"""Mixin to implement commands taking a package specification.
@@ -266,7 +267,7 @@ def customize_parser(self, parser, subparsers,
return subp
- def get_spec(self, _can_be_local=False):
+ def get_spec(self, _can_be_local=False, _can_be_url=False):
"""
Return the package specification requested.
@@ -284,6 +285,10 @@ def get_spec(self, _can_be_local=False):
raise PgxnClientException(
_("you cannot use a local resource with this command"))
+ if not _can_be_url and spec.is_url():
+ raise PgxnClientException(
+ _("you cannot use an url with this command"))
+
return spec
def get_best_version(self, data, spec, quiet=False):
@@ -360,7 +365,7 @@ def get_meta(self, spec):
Return the object obtained parsing the JSON.
"""
- if not spec.is_local():
+ if spec.is_name():
# Get the metadata from the API
try:
data = self.api.dist(spec.name)
@@ -385,9 +390,18 @@ def get_meta(self, spec):
return load_json(f)
elif spec.is_file():
- # Get the metadata from a zip file
- return get_meta_from_zip(spec.filename)
+ arc = archive.from_spec(spec)
+ return arc.get_meta()
+ elif spec.is_url():
+ with network.get_file(spec.url) as fin:
+ with temp_dir() as dir:
+ fn = network.download(fin, dir)
+ arc = archive.from_file(fn)
+ return arc.get_meta()
+
+ else:
+ assert False
class WithSpecLocal(WithSpec):
"""
@@ -406,35 +420,30 @@ def customize_parser(self, parser, subparsers, epilog=None, **kwargs):
return subp
- def get_spec(self):
- return super(WithSpecLocal, self).get_spec(_can_be_local=True)
-
+ def get_spec(self, **kwargs):
+ kwargs['_can_be_local'] = True
+ return super(WithSpecLocal, self).get_spec(**kwargs)
-import shutil
-import tempfile
-from pgxnclient.utils.zip import unpack
-class WithUnpacking(object):
+class WithSpecUrl(WithSpec):
"""
- Mixin to implement commands that may deal with zip files.
+ Mixin to implement commands that can also refer to a URL.
"""
- def call_with_temp_dir(self, f, *args, **kwargs):
- """
- Call a function in the context of a temporary directory.
- Create the temp directory and pass its name as first argument to *f*.
- Other arguments and keywords are passed to *f* too. Upon exit delete
- the directory.
- """
- dir = tempfile.mkdtemp()
- try:
- return f(dir, *args, **kwargs)
- finally:
- shutil.rmtree(dir)
+ @classmethod
+ def customize_parser(self, parser, subparsers, epilog=None, **kwargs):
+ epilog = _("""
+SPEC may also be an url specifying a protocol such as 'http://' or 'https://'.
+""") + (epilog or "")
- def unpack(self, zipname, destdir):
- """Unpack the zip file *zipname* into *destdir*."""
- return unpack(zipname, destdir)
+ subp = super(WithSpecUrl, self).customize_parser(
+ parser, subparsers, epilog=epilog, **kwargs)
+
+ return subp
+
+ def get_spec(self, **kwargs):
+ kwargs['_can_be_url'] = True
+ return super(WithSpecUrl, self).get_spec(**kwargs)
class WithPgConfig(object):
@@ -449,8 +458,8 @@ def customize_parser(self, parser, subparsers, **kwargs):
subp = super(WithPgConfig, self).customize_parser(
parser, subparsers, **kwargs)
- subp.add_argument('--pg_config', metavar="PATH", default='pg_config',
- help = _("path to the pg_config executable to find the database"
+ subp.add_argument('--pg_config', metavar="PROG", default='pg_config',
+ help = _("the pg_config executable to find the database"
" [default: %(default)s]"))
return subp
@@ -482,24 +491,33 @@ def get_pg_config(self):
if os.path.split(pg_config)[0]:
pg_config = os.path.abspath(pg_config)
else:
- for dir in os.environ.get('PATH', '').split(os.pathsep):
- if not dir: continue
- fn = os.path.abspath(os.path.join(dir, pg_config))
- if os.path.exists(fn):
- pg_config = os.path.abspath(fn)
- break
- else:
- raise PgxnClientException(_("pg_config executable not found"))
-
+ pg_config = find_executable(pg_config)
+ if not pg_config:
+ raise PgxnClientException(_("pg_config executable not found"))
return pg_config
import shlex
-class WithMake(WithPgConfig, WithUnpacking):
+class WithMake(WithPgConfig):
"""
Mixin to implement commands that should invoke :program:`make`.
"""
+ @classmethod
+ def customize_parser(self, parser, subparsers, **kwargs):
+ """
+ Add the ``--make`` option to the options parser.
+ """
+ subp = super(WithMake, self).customize_parser(
+ parser, subparsers, **kwargs)
+
+ subp.add_argument('--make', metavar="PROG",
+ default=self._find_default_make(),
+ help = _("the 'make' executable to use to build the extension "
+ "[default: %(default)s]"))
+
+ return subp
+
def run_make(self, cmd, dir, env=None, sudo=None):
"""Invoke make with the selected command.
@@ -522,7 +540,7 @@ def run_make(self, cmd, dir, env=None, sudo=None):
if sudo:
cmdline.extend(shlex.split(sudo))
- cmdline.extend(['make', 'PG_CONFIG=%s' % self.get_pg_config()])
+ cmdline.extend([self.get_make(), 'PG_CONFIG=%s' % self.get_pg_config()])
if isinstance(cmd, basestring):
cmdline.append(cmd)
@@ -536,6 +554,48 @@ def run_make(self, cmd, dir, env=None, sudo=None):
raise ProcessError(_("command returned %s: %s")
% (p.returncode, ' '.join(cmdline)))
+ def get_make(self, _cache=[]):
+ """
+ Return the path of the make binary.
+ """
+ # the cache is not for performance but to return a consistent value
+ # even if the cwd is changed
+ if _cache:
+ return _cache[0]
+
+ make = self.opts.make
+
+ if os.path.split(make)[0]:
+ # At least a relative dir specified.
+ if not os.path.exists(make):
+ raise PgxnClientException(_("make executable not found: %s")
+ % make)
+
+ # Convert to abs path to be robust in case the dir is changed.
+ make = os.path.abspath(make)
+
+ else:
+ # we don't find make here and convert to abs path because it's a
+ # security hole: make may be run under sudo and in this case we
+ # don't want root to execute a make hacked in an user local dir
+ if not find_executable(make):
+ raise PgxnClientException(_("make executable not found: %s")
+ % make)
+
+ _cache.append(make)
+ return make
+
+ @classmethod
+ def _find_default_make(self):
+ for make in ('gmake', 'make'):
+ path = find_executable(make)
+ if path:
+ return make
+
+ # if nothing was found, fall back on 'gmake'. If it was missing we
+ # will give an error when attempting to use it
+ return 'gmake'
+
class WithSudo(object):
"""
View
48 pgxnclient/commands/install.py
@@ -17,18 +17,20 @@
from subprocess import PIPE
from pgxnclient import SemVer
+from pgxnclient import archive
+from pgxnclient import network
from pgxnclient.i18n import _, N_
from pgxnclient.utils import sha1, b
from pgxnclient.errors import BadChecksum, PgxnClientException, InsufficientPrivileges
-from pgxnclient.network import download
from pgxnclient.commands import Command, WithDatabase, WithMake, WithPgConfig
-from pgxnclient.commands import WithSpec, WithSpecLocal, WithSudo
+from pgxnclient.commands import WithSpecUrl, WithSpecLocal, WithSudo
+from pgxnclient.utils.temp import temp_dir
from pgxnclient.utils.strings import Identifier
logger = logging.getLogger('pgxnclient.commands')
-class Download(WithSpec, Command):
+class Download(WithSpecUrl, Command):
name = 'download'
description = N_("download a distribution from the network")
@@ -43,6 +45,11 @@ def customize_parser(self, parser, subparsers, **kwargs):
def run(self):
spec = self.get_spec()
+ assert not spec.is_local()
+
+ if spec.is_url():
+ return self._run_url(spec)
+
data = self.get_meta(spec)
try:
@@ -52,11 +59,17 @@ def run(self):
"sha1 missing from the distribution meta")
with self.api.download(data['name'], SemVer(data['version'])) as fin:
- fn = self._get_local_file_name(fin.url)
- fn = download(fin, fn, rename=True)
+ fn = network.download(fin, self.opts.target)
+
self.verify_checksum(fn, chk)
return fn
+ def _run_url(self, spec):
+ with network.get_file(spec.url) as fin:
+ fn = network.download(fin, self.opts.target)
+
+ return fn
+
def verify_checksum(self, fn, chk):
"""Verify that a downloaded file has the expected sha1."""
sha = sha1()
@@ -77,34 +90,27 @@ def verify_checksum(self, fn, chk):
fn, sha, chk)
raise BadChecksum(_("bad sha1 in downloaded file"))
- def _get_local_file_name(self, url):
- from urlparse import urlsplit
- if os.path.isdir(self.opts.target):
- basename = urlsplit(url)[2].rsplit('/', 1)[-1]
- fn = os.path.join(self.opts.target, basename)
- else:
- fn = self.opts.target
-
- return os.path.abspath(fn)
-
-class InstallUninstall(WithMake, WithSpecLocal, Command):
+class InstallUninstall(WithMake, WithSpecUrl, WithSpecLocal, Command):
"""
Base class to implement the ``install`` and ``uninstall`` commands.
"""
def run(self):
- return self.call_with_temp_dir(self._run)
+ with temp_dir() as dir:
+ return self._run(dir)
def _run(self, dir):
spec = self.get_spec()
if spec.is_dir():
pdir = os.path.abspath(spec.dirname)
elif spec.is_file():
- pdir = self.unpack(spec.filename, dir)
- else: # download
+ pdir = archive.from_file(spec.filename).unpack(dir)
+ elif not spec.is_local():
self.opts.target = dir
fn = Download(self.opts).run()
- pdir = self.unpack(fn, dir)
+ pdir = archive.from_file(fn).unpack(dir)
+ else:
+ assert False
self.maybe_run_configure(pdir)
@@ -220,7 +226,7 @@ def _inun(self, pdir):
raise
-class LoadUnload(WithPgConfig, WithDatabase, WithSpecLocal, Command):
+class LoadUnload(WithPgConfig, WithDatabase, WithSpecUrl, WithSpecLocal, Command):
"""
Base class to implement the ``load`` and ``unload`` commands.
"""
View
22 pgxnclient/network.py
@@ -8,6 +8,7 @@
import os
import urllib2
+from urlparse import urlsplit
from itertools import count
from contextlib import closing
@@ -39,16 +40,33 @@ def get_file(url):
except urllib2.URLError, e:
raise NetworkError(_("network error: %s") % e.reason)
+def get_local_file_name(target, url):
+ """Return a good name for a local file.
+
+ If *target* is a dir, make a name out of the url. Otherwise return target
+ itself. Always return an absolute path.
+ """
+ if os.path.isdir(target):
+ basename = urlsplit(url)[2].rsplit('/', 1)[-1]
+ fn = os.path.join(target, basename)
+ else:
+ fn = target
+
+ return os.path.abspath(fn)
+
def download(f, fn, rename=True):
"""Download a file locally.
- :param f: open file read
- :param fn: name of the file to write
+ :param f: open file to read
+ :param fn: name of the file to write. If a dir, save into it.
:param rename: if true and a file *fn* exist, rename the downloaded file
adding a prefix ``-1``, ``-2``... before the extension.
Return the name of the file saved.
"""
+ if os.path.isdir(fn):
+ fn = get_local_file_name(fn, f.url)
+
if rename:
if os.path.exists(fn):
base, ext = os.path.splitext(fn)
View
40 pgxnclient/spec.py
@@ -9,6 +9,7 @@
import os
import re
+import urllib
import operator as _op
from pgxnclient.i18n import _
@@ -33,26 +34,33 @@ class Spec(object):
'stable': STABLE, }
def __init__(self, name=None, op=None, ver=None,
- dirname=None, filename=None):
+ dirname=None, filename=None, url=None):
self.name = name and name.lower()
self.op = op
self.ver = ver
- # point to local files
+ # point to local files or specific resources
self.dirname = dirname
self.filename = filename
+ self.url = url
+
+ def is_name(self):
+ return self.name is not None
def is_dir(self):
return self.dirname is not None
def is_file(self):
return self.filename is not None
+ def is_url(self):
+ return self.url is not None
+
def is_local(self):
return self.is_dir() or self.is_file()
def __str__(self):
- name = self.name or self.filename or self.dirname or "???"
+ name = self.name or self.filename or self.dirname or self.url or "???"
if self.op is None:
return name
else:
@@ -64,14 +72,28 @@ def parse(self, spec):
Raise BadSpecError if couldn't parse.
"""
- if os.sep in spec:
+ # check if it's a network resource
+ if spec.startswith('http://') or spec.startswith('https://'):
+ return Spec(url=spec)
+
+ # check if it's a local resource
+ if spec.startswith('file://'):
+ try_file = urllib.unquote_plus(spec[len('file://'):])
+ elif os.sep in spec:
+ try_file = spec
+ else:
+ try_file = None
+
+ if try_file:
# This is a local thing, let's see what
- if os.path.isdir(spec):
- return Spec(dirname=spec)
- elif os.path.exists(spec):
- return Spec(filename=spec)
+ if os.path.isdir(try_file):
+ return Spec(dirname=try_file)
+ elif os.path.exists(try_file):
+ return Spec(filename=try_file)
else:
- raise ResourceNotFound(_("cannot find '%s'") % spec)
+ raise ResourceNotFound(_("cannot find '%s'") % try_file)
+
+ # so we think it's a PGXN spec
# split operator/version and name
m = re.match(r'(.+?)(?:(==|=|>=|>|<=|<)(.*))?$', spec)
View
69 pgxnclient/tar.py
@@ -0,0 +1,69 @@
+"""
+pgxnclient -- tar file utilities
+"""
+
+# Copyright (C) 2011-2012 Daniele Varrazzo
+
+# This file is part of the PGXN client
+
+import os
+import tarfile
+
+from pgxnclient.i18n import _
+from pgxnclient.errors import PgxnClientException
+from pgxnclient.archive import Archive
+
+import logging
+logger = logging.getLogger('pgxnclient.tar')
+
+
+class TarArchive(Archive):
+ """Handle .tar archives"""
+ _file = None
+
+ def can_open(self):
+ return tarfile.is_tarfile(self.filename)
+
+ def open(self):
+ assert not self._file, "archive already open"
+ try:
+ self._file = tarfile.open(self.filename, 'r')
+ except Exception, e:
+ raise PgxnClientException(
+ _("cannot open archive '%s': %s") % (self.filename, e))
+
+ def close(self):
+ if self._file is not None:
+ self._file.close()
+ self._file = None
+
+ def list_files(self):
+ assert self._file, "archive not open"
+ return self._file.getnames()
+
+ def read(self, fn):
+ assert self._file, "archive not open"
+ return self._file.extractfile(fn).read()
+
+ def unpack(self, destdir):
+ tarname = self.filename
+ logger.info(_("unpacking: %s"), tarname)
+ destdir = os.path.abspath(destdir)
+ self.open()
+ try:
+ for fn in self.list_files():
+ fname = os.path.abspath(os.path.join(destdir, fn))
+ if not fname.startswith(destdir):
+ raise PgxnClientException(
+ _("archive file '%s' trying to escape!") % fname)
+
+ self._file.extractall(path=destdir)
+ finally:
+ self.close()
+
+ return self._find_work_directory(destdir)
+
+
+def unpack(filename, destdir):
+ return TarArchive(filename).unpack(destdir)
+
View
67 pgxnclient/tests/test_archives.py
@@ -0,0 +1,67 @@
+from pgxnclient import tar
+from pgxnclient import zip
+from pgxnclient import archive
+
+from pgxnclient.tests import unittest
+from pgxnclient.errors import PgxnClientException
+from pgxnclient.tests.testutils import get_test_filename
+
+class TestArchive(unittest.TestCase):
+ def test_from_file_zip(self):
+ fn = get_test_filename('foobar-0.42.1.zip')
+ a = archive.from_file(fn)
+ self.assert_(isinstance(a, zip.ZipArchive))
+ self.assertEqual(a.filename, fn)
+
+ def test_from_file_tar(self):
+ fn = get_test_filename('foobar-0.42.1.tar.gz')
+ a = archive.from_file(fn)
+ self.assert_(isinstance(a, tar.TarArchive))
+ self.assertEqual(a.filename, fn)
+
+ def test_from_file_unknown(self):
+ fn = get_test_filename('META-manyext.json')
+ self.assertRaises(PgxnClientException(archive.from_file, fn))
+
+
+class TestZipArchive(unittest.TestCase):
+ def test_can_open(self):
+ fn = get_test_filename('foobar-0.42.1.zip')
+ a = zip.ZipArchive(fn)
+ self.assert_(a.can_open())
+ a.open()
+ a.close()
+
+ def test_can_open_noext(self):
+ fn = get_test_filename('zip.ext')
+ a = zip.ZipArchive(fn)
+ self.assert_(a.can_open())
+ a.open()
+ a.close()
+
+ def test_cannot_open(self):
+ fn = get_test_filename('foobar-0.42.1.tar.gz')
+ a = zip.ZipArchive(fn)
+ self.assert_(not a.can_open())
+
+
+class TestTarArchive(unittest.TestCase):
+ def test_can_open(self):
+ fn = get_test_filename('foobar-0.42.1.tar.gz')
+ a = tar.TarArchive(fn)
+ self.assert_(a.can_open())
+ a.open()
+ a.close()
+
+ def test_can_open_noext(self):
+ fn = get_test_filename('tar.ext')
+ a = tar.TarArchive(fn)
+ self.assert_(a.can_open())
+ a.open()
+ a.close()
+
+ def test_cannot_open(self):
+ fn = get_test_filename('foobar-0.42.1.zip')
+ a = tar.TarArchive(fn)
+ self.assert_(not a.can_open())
+
View
254 pgxnclient/tests/test_commands.py
@@ -44,7 +44,7 @@ def f(what):
class InfoTestCase(unittest.TestCase):
def _get_output(self, cmdline):
@patch('sys.stdout')
- @patch('pgxnclient.api.get_file')
+ @patch('pgxnclient.network.get_file')
def do(mock, stdout):
mock.side_effect = fake_get_file
from pgxnclient.cli import main
@@ -120,7 +120,7 @@ def test_popen_raises(self):
class DownloadTestCase(unittest.TestCase):
- @patch('pgxnclient.api.get_file')
+ @patch('pgxnclient.network.get_file')
def test_download_latest(self, mock):
mock.side_effect = fake_get_file
@@ -134,7 +134,7 @@ def test_download_latest(self, mock):
finally:
ifunlink(fn)
- @patch('pgxnclient.api.get_file')
+ @patch('pgxnclient.network.get_file')
def test_download_testing(self, mock):
mock.side_effect = fake_get_file
@@ -148,7 +148,21 @@ def test_download_testing(self, mock):
finally:
ifunlink(fn)
- @patch('pgxnclient.api.get_file')
+ @patch('pgxnclient.network.get_file')
+ def test_download_url(self, mock):
+ mock.side_effect = fake_get_file
+
+ fn = 'foobar-0.43.2b1.zip'
+ self.assert_(not os.path.exists(fn))
+
+ from pgxnclient.cli import main
+ try:
+ main(['download', 'http://api.pgxn.org/dist/foobar/0.43.2b1/foobar-0.43.2b1.zip'])
+ self.assert_(os.path.exists(fn))
+ finally:
+ ifunlink(fn)
+
+ @patch('pgxnclient.network.get_file')
def test_download_ext(self, mock):
mock.side_effect = fake_get_file
@@ -162,7 +176,7 @@ def test_download_ext(self, mock):
finally:
ifunlink(fn)
- @patch('pgxnclient.api.get_file')
+ @patch('pgxnclient.network.get_file')
def test_download_rename(self, mock):
mock.side_effect = fake_get_file
@@ -195,7 +209,7 @@ def test_download_rename(self, mock):
ifunlink(fn1)
ifunlink(fn2)
- @patch('pgxnclient.api.get_file')
+ @patch('pgxnclient.network.get_file')
def test_download_bad_sha1(self, mock):
def fakefake(url):
return fake_get_file(url, urlmap = {
@@ -218,7 +232,7 @@ def fakefake(url):
finally:
ifunlink(fn)
- @patch('pgxnclient.api.get_file')
+ @patch('pgxnclient.network.get_file')
def test_download_case_insensitive(self, mock):
mock.side_effect = fake_get_file
@@ -307,10 +321,33 @@ def test_version(self):
self.assertEqual(res, cmd.get_best_version(data, spec))
+class Assertions(object):
+
+ make = object()
+
+ def assertCallArgs(self, pattern, args):
+ if len(pattern) != len(args):
+ self.fail('args and pattern have different lengths')
+ for p, a in zip(pattern, args):
+ if p is self.make:
+ if not a.endswith('make'):
+ self.fail('%s is not a make in %s' % (a, args))
+ else:
+ if not a == p:
+ self.fail('%s is not a %s in %s' % (a, p, args))
+
+# With mock patching a method seems tricky: looks there's no way to get to
+# 'self' as the mock method is unbound.
+from pgxnclient.tar import TarArchive
+TarArchive.unpack_orig = TarArchive.unpack
+
+from pgxnclient.zip import ZipArchive
+ZipArchive.unpack_orig = ZipArchive.unpack
+
+class InstallTestCase(unittest.TestCase, Assertions):
-class InstallTestCase(unittest.TestCase):
def setUp(self):
- self._p1 = patch('pgxnclient.api.get_file')
+ self._p1 = patch('pgxnclient.network.get_file')
self.mock_get = self._p1.start()
self.mock_get.side_effect = fake_get_file
@@ -333,8 +370,8 @@ def test_install_latest(self):
main(['install', '--sudo', '--', 'foobar'])
self.assertEquals(self.mock_popen.call_count, 2)
- self.assertEquals(['make'], self.mock_popen.call_args_list[0][0][0][:1])
- self.assertEquals(['sudo', 'make'], self.mock_popen.call_args_list[1][0][0][:2])
+ self.assertCallArgs([self.make], self.mock_popen.call_args_list[0][0][0][:1])
+ self.assertCallArgs(['sudo', self.make], self.mock_popen.call_args_list[1][0][0][:2])
def test_install_missing_sudo(self):
from pgxnclient.cli import main
@@ -348,8 +385,19 @@ def test_install_local(self):
main(['install', 'foobar'])
self.assertEquals(self.mock_popen.call_count, 2)
- self.assertEquals(['make'], self.mock_popen.call_args_list[0][0][0][:1])
- self.assertEquals(['make'], self.mock_popen.call_args_list[1][0][0][:1])
+ self.assertCallArgs([self.make], self.mock_popen.call_args_list[0][0][0][:1])
+ self.assertCallArgs([self.make], self.mock_popen.call_args_list[1][0][0][:1])
+
+ def test_install_url(self):
+ self.mock_pgconfig.side_effect = fake_pg_config(
+ libdir=os.environ['HOME'], bindir='/')
+
+ from pgxnclient.cli import main
+ main(['install', 'http://api.pgxn.org/dist/foobar/0.42.1/foobar-0.42.1.zip'])
+
+ self.assertEquals(self.mock_popen.call_count, 2)
+ self.assertCallArgs([self.make], self.mock_popen.call_args_list[0][0][0][:1])
+ self.assertCallArgs([self.make], self.mock_popen.call_args_list[1][0][0][:1])
def test_install_fails(self):
self.mock_popen.return_value.returncode = 1
@@ -381,43 +429,73 @@ def test_install_nosudo(self):
main(['install', '--nosudo', 'foobar'])
self.assertEquals(self.mock_popen.call_count, 2)
- self.assertEquals(['make'], self.mock_popen.call_args_list[0][0][0][:1])
- self.assertEquals(['make'], self.mock_popen.call_args_list[1][0][0][:1])
+ self.assertCallArgs([self.make], self.mock_popen.call_args_list[0][0][0][:1])
+ self.assertCallArgs([self.make], self.mock_popen.call_args_list[1][0][0][:1])
def test_install_sudo(self):
from pgxnclient.cli import main
main(['install', '--sudo', 'gksudo -d "hello world"', 'foobar'])
self.assertEquals(self.mock_popen.call_count, 2)
- self.assertEquals(['make'], self.mock_popen.call_args_list[0][0][0][:1])
- self.assertEquals(['gksudo', '-d', 'hello world', 'make'],
+ self.assertCallArgs([self.make],
+ self.mock_popen.call_args_list[0][0][0][:1])
+ self.assertCallArgs(['gksudo', '-d', 'hello world', self.make],
self.mock_popen.call_args_list[1][0][0][:4])
- @patch('pgxnclient.commands.unpack')
+ @patch('pgxnclient.tar.TarArchive.unpack')
+ def test_install_local_tar(self, mock_unpack):
+ fn = get_test_filename('foobar-0.42.1.tar.gz')
+ mock_unpack.side_effect = TarArchive(fn).unpack_orig
+
+ from pgxnclient.cli import main
+ main(['install', '--sudo', '--', fn])
+
+ self.assertEquals(self.mock_popen.call_count, 2)
+ self.assertCallArgs([self.make], self.mock_popen.call_args_list[0][0][0][:1])
+ self.assertCallArgs(['sudo', self.make],
+ self.mock_popen.call_args_list[1][0][0][:2])
+ make_cwd = self.mock_popen.call_args_list[1][1]['cwd']
+
+ self.assertEquals(mock_unpack.call_count, 1)
+ tmpdir, = mock_unpack.call_args[0]
+ self.assertEqual(make_cwd, os.path.join(tmpdir, 'foobar-0.42.1'))
+
+ @patch('pgxnclient.zip.ZipArchive.unpack')
def test_install_local_zip(self, mock_unpack):
- from pgxnclient.utils.zip import unpack
- mock_unpack.side_effect = unpack
+ fn = get_test_filename('foobar-0.42.1.zip')
+ mock_unpack.side_effect = ZipArchive(fn).unpack_orig
from pgxnclient.cli import main
- main(['install', '--sudo', '--', get_test_filename('foobar-0.42.1.zip')])
+ main(['install', '--sudo', '--', fn])
self.assertEquals(self.mock_popen.call_count, 2)
- self.assertEquals(['make'], self.mock_popen.call_args_list[0][0][0][:1])
- self.assertEquals(['sudo', 'make'],
+ self.assertCallArgs([self.make], self.mock_popen.call_args_list[0][0][0][:1])
+ self.assertCallArgs(['sudo', self.make],
self.mock_popen.call_args_list[1][0][0][:2])
make_cwd = self.mock_popen.call_args_list[1][1]['cwd']
self.assertEquals(mock_unpack.call_count, 1)
- zipname, tmpdir = mock_unpack.call_args[0]
- self.assertEqual(zipname, get_test_filename('foobar-0.42.1.zip'))
+ tmpdir, = mock_unpack.call_args[0]
self.assertEqual(make_cwd, os.path.join(tmpdir, 'foobar-0.42.1'))
+ def test_install_url_file(self):
+ fn = get_test_filename('foobar-0.42.1.zip')
+ url = 'file://' + os.path.abspath(fn).replace("f", '%%%2x' % ord('f'))
+
+ from pgxnclient.cli import main
+ main(['install', '--sudo', '--', url])
+
+ self.assertEquals(self.mock_popen.call_count, 2)
+ self.assertCallArgs([self.make], self.mock_popen.call_args_list[0][0][0][:1])
+ self.assertCallArgs(['sudo', self.make],
+ self.mock_popen.call_args_list[1][0][0][:2])
+
def test_install_local_dir(self):
self.mock_get.side_effect = lambda *args: self.fail('network invoked')
tdir = tempfile.mkdtemp()
try:
- from pgxnclient.utils.zip import unpack
+ from pgxnclient.zip import unpack
dir = unpack(get_test_filename('foobar-0.42.1.zip'), tdir)
from pgxnclient.cli import main
@@ -427,16 +505,16 @@ def test_install_local_dir(self):
shutil.rmtree(tdir)
self.assertEquals(self.mock_popen.call_count, 2)
- self.assertEquals(['make'], self.mock_popen.call_args_list[0][0][0][:1])
- self.assertEquals(dir, self.mock_popen.call_args_list[0][1]['cwd'])
- self.assertEquals(['sudo', 'make'],
+ self.assertCallArgs([self.make], self.mock_popen.call_args_list[0][0][0][:1])
+ self.assertCallArgs(dir, self.mock_popen.call_args_list[0][1]['cwd'])
+ self.assertCallArgs(['sudo', self.make],
self.mock_popen.call_args_list[1][0][0][:2])
self.assertEquals(dir, self.mock_popen.call_args_list[1][1]['cwd'])
-class CheckTestCase(unittest.TestCase):
+class CheckTestCase(unittest.TestCase, Assertions):
def setUp(self):
- self._p1 = patch('pgxnclient.api.get_file')
+ self._p1 = patch('pgxnclient.network.get_file')
self.mock_get = self._p1.start()
self.mock_get.side_effect = fake_get_file
@@ -459,7 +537,14 @@ def test_check_latest(self):
main(['check', 'foobar'])
self.assertEquals(self.mock_popen.call_count, 1)
- self.assertEquals(['make'], self.mock_popen.call_args_list[0][0][0][:1])
+ self.assertCallArgs([self.make], self.mock_popen.call_args_list[0][0][0][:1])
+
+ def test_check_url(self):
+ from pgxnclient.cli import main
+ main(['check', 'http://api.pgxn.org/dist/foobar/0.42.1/foobar-0.42.1.zip'])
+
+ self.assertEquals(self.mock_popen.call_count, 1)
+ self.assertCallArgs([self.make], self.mock_popen.call_args_list[0][0][0][:1])
def test_check_fails(self):
self.mock_popen.return_value.returncode = 1
@@ -542,7 +627,7 @@ def test_parse_version(self):
'PostgreSQL 9.1alpha5 on i686-pc-linux-gnu, compiled by GCC gcc'
' (Ubuntu/Linaro 4.4.4-14ubuntu5) 4.4.5, 32-bit '))
- @patch('pgxnclient.api.get_file')
+ @patch('pgxnclient.network.get_file')
def test_check_psql_options(self, mock_get):
mock_get.side_effect = fake_get_file
@@ -564,12 +649,11 @@ def test_check_psql_options(self, mock_get):
args = self.mock_popen.call_args[0][0]
self.assertEqual('somewhere', args[args.index('--host') + 1])
- @patch('pgxnclient.commands.unpack')
- @patch('pgxnclient.api.get_file')
+ @patch('pgxnclient.zip.ZipArchive.unpack')
+ @patch('pgxnclient.network.get_file')
def test_load_local_zip(self, mock_get, mock_unpack):
mock_get.side_effect = lambda *args: self.fail('network invoked')
- from pgxnclient.utils.zip import unpack
- mock_unpack.side_effect = unpack
+ mock_unpack.side_effect = ZipArchive.unpack_orig
from pgxnclient.cli import main
main(['load', '--yes', get_test_filename('foobar-0.42.1.zip')])
@@ -579,15 +663,31 @@ def test_load_local_zip(self, mock_get, mock_unpack):
self.assert_('psql' in self.mock_popen.call_args[0][0][0])
communicate = self.mock_popen.return_value.communicate
self.assertEquals(communicate.call_args[0][0],
- 'CREATE EXTENSION foobar;')
+ b('CREATE EXTENSION foobar;'))
+
+ @patch('pgxnclient.tar.TarArchive.unpack')
+ @patch('pgxnclient.network.get_file')
+ def test_load_local_tar(self, mock_get, mock_unpack):
+ mock_get.side_effect = lambda *args: self.fail('network invoked')
+ mock_unpack.side_effect = TarArchive.unpack_orig
+
+ from pgxnclient.cli import main
+ main(['load', '--yes', get_test_filename('foobar-0.42.1.tar.gz')])
+
+ self.assertEquals(mock_unpack.call_count, 0)
+ self.assertEquals(self.mock_popen.call_count, 1)
+ self.assert_('psql' in self.mock_popen.call_args[0][0][0])
+ communicate = self.mock_popen.return_value.communicate
+ self.assertEquals(communicate.call_args[0][0],
+ b('CREATE EXTENSION foobar;'))
- @patch('pgxnclient.api.get_file')
+ @patch('pgxnclient.network.get_file')
def test_load_local_dir(self, mock_get):
mock_get.side_effect = lambda *args: self.fail('network invoked')
tdir = tempfile.mkdtemp()
try:
- from pgxnclient.utils.zip import unpack
+ from pgxnclient.zip import unpack
dir = unpack(get_test_filename('foobar-0.42.1.zip'), tdir)
from pgxnclient.cli import main
@@ -600,12 +700,46 @@ def test_load_local_dir(self, mock_get):
self.assert_('psql' in self.mock_popen.call_args[0][0][0])
communicate = self.mock_popen.return_value.communicate
self.assertEquals(communicate.call_args[0][0],
- 'CREATE EXTENSION foobar;')
+ b('CREATE EXTENSION foobar;'))
+
+ @patch('pgxnclient.zip.ZipArchive.unpack')
+ @patch('pgxnclient.network.get_file')
+ def test_load_zip_url(self, mock_get, mock_unpack):
+ mock_get.side_effect = fake_get_file
+ mock_unpack.side_effect = ZipArchive.unpack_orig
+
+ from pgxnclient.cli import main
+ main(['load', '--yes',
+ 'http://api.pgxn.org/dist/foobar/0.42.1/foobar-0.42.1.zip'])
+
+ self.assertEquals(mock_unpack.call_count, 0)
+ self.assertEquals(self.mock_popen.call_count, 1)
+ self.assert_('psql' in self.mock_popen.call_args[0][0][0])
+ communicate = self.mock_popen.return_value.communicate
+ self.assertEquals(communicate.call_args[0][0],
+ b('CREATE EXTENSION foobar;'))
+
+ @patch('pgxnclient.tar.TarArchive.unpack')
+ @patch('pgxnclient.network.get_file')
+ def test_load_tar_url(self, mock_get, mock_unpack):
+ mock_get.side_effect = fake_get_file
+ mock_unpack.side_effect = TarArchive.unpack_orig
+
+ from pgxnclient.cli import main
+ main(['load', '--yes',
+ 'http://example.org/foobar-0.42.1.tar.gz'])
+
+ self.assertEquals(mock_unpack.call_count, 0)
+ self.assertEquals(self.mock_popen.call_count, 1)
+ self.assert_('psql' in self.mock_popen.call_args[0][0][0])
+ communicate = self.mock_popen.return_value.communicate
+ self.assertEquals(communicate.call_args[0][0],
+ b('CREATE EXTENSION foobar;'))
def test_load_extensions_order(self):
tdir = tempfile.mkdtemp()
try:
- from pgxnclient.utils.zip import unpack
+ from pgxnclient.zip import unpack
dir = unpack(get_test_filename('foobar-0.42.1.zip'), tdir)
shutil.copyfile(
get_test_filename('META-manyext.json'),
@@ -621,18 +755,18 @@ def test_load_extensions_order(self):
self.assert_('psql' in self.mock_popen.call_args[0][0][0])
communicate = self.mock_popen.return_value.communicate
self.assertEquals(communicate.call_args_list[0][0][0],
- 'CREATE EXTENSION foo;')
+ b('CREATE EXTENSION foo;'))
self.assertEquals(communicate.call_args_list[1][0][0],
- 'CREATE EXTENSION bar;')
+ b('CREATE EXTENSION bar;'))
self.assertEquals(communicate.call_args_list[2][0][0],
- 'CREATE EXTENSION baz;')
+ b('CREATE EXTENSION baz;'))
self.assertEquals(communicate.call_args_list[3][0][0],
- 'CREATE EXTENSION qux;')
+ b('CREATE EXTENSION qux;'))
def test_unload_extensions_order(self):
tdir = tempfile.mkdtemp()
try:
- from pgxnclient.utils.zip import unpack
+ from pgxnclient.zip import unpack
dir = unpack(get_test_filename('foobar-0.42.1.zip'), tdir)
shutil.copyfile(
get_test_filename('META-manyext.json'),
@@ -648,18 +782,18 @@ def test_unload_extensions_order(self):
self.assert_('psql' in self.mock_popen.call_args[0][0][0])
communicate = self.mock_popen.return_value.communicate
self.assertEquals(communicate.call_args_list[0][0][0],
- 'DROP EXTENSION qux;')
+ b('DROP EXTENSION qux;'))
self.assertEquals(communicate.call_args_list[1][0][0],
- 'DROP EXTENSION baz;')
+ b('DROP EXTENSION baz;'))
self.assertEquals(communicate.call_args_list[2][0][0],
- 'DROP EXTENSION bar;')
+ b('DROP EXTENSION bar;'))
self.assertEquals(communicate.call_args_list[3][0][0],
- 'DROP EXTENSION foo;')
+ b('DROP EXTENSION foo;'))
def test_load_list(self):
tdir = tempfile.mkdtemp()
try:
- from pgxnclient.utils.zip import unpack
+ from pgxnclient.zip import unpack
dir = unpack(get_test_filename('foobar-0.42.1.zip'), tdir)
shutil.copyfile(
get_test_filename('META-manyext.json'),
@@ -675,14 +809,14 @@ def test_load_list(self):
self.assert_('psql' in self.mock_popen.call_args[0][0][0])
communicate = self.mock_popen.return_value.communicate
self.assertEquals(communicate.call_args_list[0][0][0],
- 'CREATE EXTENSION baz;')
+ b('CREATE EXTENSION baz;'))
self.assertEquals(communicate.call_args_list[1][0][0],
- 'CREATE EXTENSION foo;')
+ b('CREATE EXTENSION foo;'))
def test_unload_list(self):
tdir = tempfile.mkdtemp()
try:
- from pgxnclient.utils.zip import unpack
+ from pgxnclient.zip import unpack
dir = unpack(get_test_filename('foobar-0.42.1.zip'), tdir)
shutil.copyfile(
get_test_filename('META-manyext.json'),
@@ -698,14 +832,14 @@ def test_unload_list(self):
self.assert_('psql' in self.mock_popen.call_args[0][0][0])
communicate = self.mock_popen.return_value.communicate
self.assertEquals(communicate.call_args_list[0][0][0],
- 'DROP EXTENSION baz;')
+ b('DROP EXTENSION baz;'))
self.assertEquals(communicate.call_args_list[1][0][0],
- 'DROP EXTENSION foo;')
+ b('DROP EXTENSION foo;'))
def test_load_missing(self):
tdir = tempfile.mkdtemp()
try:
- from pgxnclient.utils.zip import unpack
+ from pgxnclient.zip import unpack
dir = unpack(get_test_filename('foobar-0.42.1.zip'), tdir)
shutil.copyfile(
get_test_filename('META-manyext.json'),
@@ -723,7 +857,7 @@ def test_load_missing(self):
def test_unload_missing(self):
tdir = tempfile.mkdtemp()
try:
- from pgxnclient.utils.zip import unpack
+ from pgxnclient.zip import unpack
dir = unpack(get_test_filename('foobar-0.42.1.zip'), tdir)
shutil.copyfile(
get_test_filename('META-manyext.json'),
@@ -741,7 +875,7 @@ def test_unload_missing(self):
class SearchTestCase(unittest.TestCase):
@patch('sys.stdout')
- @patch('pgxnclient.api.get_file')
+ @patch('pgxnclient.network.get_file')
def test_search_quoting(self, mock_get, stdout):
mock_get.side_effect = fake_get_file
from pgxnclient.cli import main
View
17 pgxnclient/utils/__init__.py
@@ -7,10 +7,12 @@
# This file is part of the PGXN client
-__all__ = ['OrderedDict', 'load_json', 'load_jsons', 'sha1', 'b']
+__all__ = ['OrderedDict', 'load_json', 'load_jsons', 'sha1', 'b',
+ 'find_executable']
import sys
+import os
# OrderedDict available from Python 2.7
if sys.version_info >= (2, 7):
@@ -54,3 +56,16 @@ def b(s):
def b(s):
return s.encode('utf8')
+
+def find_executable(name):
+ """
+ Find executable by ``name`` by inspecting PATH environment variable, return
+ ``None`` if nothing found.
+ """
+ for dir in os.environ.get('PATH', '').split(os.pathsep):
+ if not dir:
+ continue
+ fn = os.path.abspath(os.path.join(dir, name))
+ if os.path.exists(fn):
+ return os.path.abspath(fn)
+
View
19 pgxnclient/utils/temp.py
@@ -0,0 +1,19 @@
+"""
+pgxnclient -- temp files utilities
+"""
+
+# Copyright (C) 2011-2012 Daniele Varrazzo
+
+# This file is part of the PGXN client
+
+import shutil
+import tempfile
+import contextlib
+
+@contextlib.contextmanager
+def temp_dir():
+ """Context manager to create a temp dir and delete after usage."""
+ dir = tempfile.mkdtemp()
+ yield dir
+ shutil.rmtree(dir)
+
View
88 pgxnclient/utils/zip.py
@@ -1,88 +0,0 @@
-"""
-pgxnclient -- zip file utilities
-"""
-
-# Copyright (C) 2011-2012 Daniele Varrazzo
-
-# This file is part of the PGXN client
-
-import os
-import stat
-from zipfile import ZipFile
-
-from pgxnclient.utils import b, load_jsons
-from pgxnclient.i18n import _
-from pgxnclient.errors import PgxnClientException
-
-import logging
-logger = logging.getLogger('pgxnclient.utils.zip')
-
-def unpack(zipname, destdir):
- logger.info(_("unpacking: %s"), zipname)
- destdir = os.path.abspath(destdir)
- zf = ZipFile(zipname, 'r')
- try:
- for fn in zf.namelist():
- fname = os.path.abspath(os.path.join(destdir, fn))
- if not fname.startswith(destdir):
- raise PgxnClientException(
- _("archive file '%s' trying to escape!") % fname)
-
- # Looks like checking for a trailing / is the only way to
- # tell if the file is a directory.
- if fn.endswith('/'):
- os.makedirs(fname)
- continue
-
- # The directory is not always explicitly present in the archive
- if not os.path.exists(os.path.dirname(fname)):
- os.makedirs(os.path.dirname(fname))
-
- # Copy the file content
- logger.debug(_("saving: %s"), fname)
- fout = open(fname, "wb")
- try:
- data = zf.read(fn)
- # In order to restore the executable bit, I haven't find
- # anything that looks like an executable flag in the zipinfo,
- # so look at the hashbangs...
- isexec = data[:2] == b('#!')
- fout.write(data)
- finally:
- fout.close()
-
- if isexec:
- os.chmod(fname, stat.S_IREAD | stat.S_IWRITE | stat.S_IEXEC)
-
- finally:
- zf.close()
-
- # Choose the directory where to work. Because we are mostly a wrapper for
- # pgxs, let's look for a makefile. The zip should contain a single base
- # directory, so return the first dir we found containing a Makefile,
- # alternatively just return the unpacked dir
- for dir in os.listdir(destdir):
- for fn in ('Makefile', 'makefile', 'GNUmakefile', 'configure'):
- if os.path.exists(os.path.join(destdir, dir, fn)):
- return os.path.join(destdir, dir)
-
- return destdir
-
-def get_meta_from_zip(filename):
- try:
- zf = ZipFile(filename, 'r')
- except Exception, e:
- raise PgxnClientException(
- _("cannot open archive '%s': %s") % (filename, e))
-
- try:
- # Return the first file with the expected name
- for fn in zf.namelist():
- if fn.endswith('META.json'):
- return load_jsons(zf.read(fn).decode('utf8'))
- else:
- raise PgxnClientException(
- _("file 'META.json' not found in archive '%s'") % filename)
- finally:
- zf.close()
-
View
97 pgxnclient/zip.py
@@ -0,0 +1,97 @@
+"""
+pgxnclient -- zip file utilities
+"""
+
+# Copyright (C) 2011-2012 Daniele Varrazzo
+
+# This file is part of the PGXN client
+
+import os
+import stat
+import zipfile
+
+from pgxnclient.utils import b
+from pgxnclient.i18n import _
+from pgxnclient.errors import PgxnClientException
+from pgxnclient.archive import Archive
+
+import logging
+logger = logging.getLogger('pgxnclient.zip')
+
+
+class ZipArchive(Archive):
+ """Handle .zip archives"""
+
+ _file = None
+
+ def can_open(self):
+ return zipfile.is_zipfile(self.filename)
+
+ def open(self):
+ assert not self._file, "archive already open"
+ try:
+ self._file = zipfile.ZipFile(self.filename, 'r')
+ except Exception, e:
+ raise PgxnClientException(
+ _("cannot open archive '%s': %s") % (self.filename, e))
+
+ def close(self):
+ if self._file is not None:
+ self._file.close()
+ self._file = None
+
+ def list_files(self):
+ assert self._file, "archive not open"
+ return self._file.namelist()
+
+ def read(self, fn):
+ assert self._file, "archive not open"
+ return self._file.read(fn)
+
+ def unpack(self, destdir):
+ zipname = self.filename
+ logger.info(_("unpacking: %s"), zipname)
+ destdir = os.path.abspath(destdir)
+ self.open()
+ try:
+ for fn in self.list_files():
+ fname = os.path.abspath(os.path.join(destdir, fn))
+ if not fname.startswith(destdir):
+ raise PgxnClientException(
+ _("archive file '%s' trying to escape!") % fname)
+
+ # Looks like checking for a trailing / is the only way to
+ # tell if the file is a directory.
+ if fn.endswith('/'):
+ os.makedirs(fname)
+ continue
+
+ # The directory is not always explicitly present in the archive
+ if not os.path.exists(os.path.dirname(fname)):
+ os.makedirs(os.path.dirname(fname))
+
+ # Copy the file content
+ logger.debug(_("saving: %s"), fname)
+ fout = open(fname, "wb")
+ try:
+ data = self.read(fn)
+ # In order to restore the executable bit, I haven't find
+ # anything that looks like an executable flag in the zipinfo,
+ # so look at the hashbangs...
+ isexec = data[:2] == b('#!')
+ fout.write(data)
+ finally:
+ fout.close()
+
+ if isexec:
+ os.chmod(fname, stat.S_IREAD | stat.S_IWRITE | stat.S_IEXEC)
+
+ finally:
+ self.close()
+
+ return self._find_work_directory(destdir)
+
+
+def unpack(filename, destdir):
+ return ZipArchive(filename).unpack(destdir)
+
View
BIN testdata/foobar-0.42.1.tar.gz
Binary file not shown.
View
1 testdata/http%3A%2F%2Fexample.org%2Ffoobar-0.42.1.tar.gz
View
1 testdata/tar.ext
View
1 testdata/zip.ext

0 comments on commit 3e33bb1

Please sign in to comment.
Something went wrong with that request. Please try again.