invenio-devscripts repository contains a collection of scripts
useful when hacking on Invenio.
- invenio-make-install
- install (parts of) source tree and restart Invenio WSGI application
- invenio-check-kwalitee
- check code kwalitee of source files
- invenio-check-branch
- check code kwalitee changes between branches
- invenio-recreate-demo-site
- recreate demo site
- invenio-retest-demo-site
- rerun test suite on recreated demo site
- invenio-initialise-virtualenv
- initialise virtualenv for Invenio development
- invenio-backup-site
- create backup file for the given Invenio site instance
- invenio-restore-site
- restore Invenio site instance from the backup file
- invenio-create-deploy-recipe
- create deploy recipes
- invenio-check-AUTHORS
- check whether all committers are well present in AUTHORS or THANKS files
- invenio-check-POTFILES
- check whether all sources files that contain I18N phrases are well present in POTFILES.in
- invenio-delete-old-git-tags
- delete old CVS-times tags from git repository
- invenio-code-browser
- generate and publish Invenio code browser pages
Put the scripts somewhere under your PATH. For example, if you
clone this repository under $HOME/private/src:
cd $HOME/private/src git clone https://github.com/tiborsimko/invenio-devscripts.git
then in your $HOME/.bashrc you can say:
export PATH=$HOME/private/src/invenio-devscripts:$PATH
You will need to configure some of the following environment variables:
| Variable | Documentation | Default value |
| CFG_INVENIO_SRCDIR | where are Invenio sources? | ~/private/src/invenio |
| CFG_INVENIO_PREFIX | where is Invenio installed? | /opt/invenio |
| CFG_INVENIO_HOSTNAME | what is allowed hostname? | pcuds07 |
| CFG_INVENIO_DOMAINNAME | what is our domain name? | cern.ch |
| CFG_INVENIO_PORT_HTTP | what is our HTTP port number? | 80 |
| CFG_INVENIO_PORT_HTTPS | what is our HTTPS port number? | 443 |
| CFG_INVENIO_USER | under which user ID Invenio runs? | www-data |
| CFG_INVENIO_ADMIN | who is admin of this instance? | tibor.simko@cern.ch |
| CFG_INVENIO_DATABASE_NAME | what is database name? | invenio |
| CFG_INVENIO_DATABASE_USER | what is database user? | invenio |
| CFG_INVENIO_APACHECTL | how to restart Apache? | /etc/init.d/apache2 |
| CFG_INVENIO_MYSQLCTL | how to restart MySQL? | /etc/init.d/mysql |
| CFG_INVENIO_VIRTUALENVS | where is virtualenv home? | ~/.virtualenvs |
Here is a minimal example of what you can put in your $HOME/.bashrc:
export CFG_INVENIO_HOSTNAME=doc export CFG_INVENIO_DOMAINNAME=example.org export CFG_INVENIO_ADMIN=john.doe@example.org
Note that if you don’t set up CFG variables, you can still call devscripts “on the spot”, for example:
CFG_INVENIO_HOSTNAME=newbox CFG_INVENIO_USER=apache invenio-retest-demo-site --yes-i-know
Note that devscripts further assume that you have a running Invenio instance installed on your box. If you do not have Invenio installed yet, then follow pages like Invenio on Debian; it takes only a few minutes to install Invenio for the first time.
Some devscripts, especially ones that recreate Invenio demo site for
you, further assume that you have certain sudo rights to execute
certain commands:
$ cat /etc/sudoers.d/johndoe
johndoe ALL=(www-data) NOPASSWD: ALL, \
(root) NOPASSWD: /bin/rm -rf /opt/invenio/var/tmp/ooffice-tmp-files, \
(root) NOPASSWD: /bin/mkdir -p /opt/invenio/var/tmp/ooffice-tmp-files, \
(root) NOPASSWD: /bin/chown -R nobody /opt/invenio/var/tmp/ooffice-tmp-files, \
(root) NOPASSWD: /bin/chmod -R 755 /opt/invenio/var/tmp/ooffice-tmp-files, \
(root) NOPASSWD: /etc/init.d/apache2, \
(root) NOPASSWD: /etc/init.d/mysql
This is the most frequently used devscript. Depending on where in the Invenio source tree it is called from, the devscript installs the current part of the source tree and restarts Invenio WSGI application. For example, when you are hacking on WebMessage Python sources files, you can deploy your changes to the running Invenio instance in no time. (Well, in 0.11 sec, on my box.)
Because this devscript is so frequently used, it is advantageous to
create shortcuts for it, for example a shell alias called mi (for
“make install”):
alias mi="$HOME/private/src/invenio-devscripts/invenio-make-install"
and a hot key for your preferred editor; an example for Emacs:
(defun tibor-invenio-make-install ()
"Launch invenio-make-install script on the current buffer."
(interactive)
(save-buffer)
(shell-command "~/private/src/invenio-devscripts/invenio-make-install"))
(global-set-key (kbd "C-c i") 'tibor-invenio-make-install)This allows you to press C-c i to install your edits.
This devscript is also very frequently used. When hacking on say
webmessage_dblayer.py, calling this script will detect the most
common code kwalitee problems:
invenio-check-kwalitee --check-some webmessage_dblayer.py
Because this devscript is so frequently used, you may again want to
create a short shell alias for it, say kw (=for “kwalitee”):
alias kw="$HOME/private/src/invenio-devscripts/invenio-check-kwalitee --check-some"
and a hot key for your preferred editor; an example for Emacs:
(defun tibor-invenio-check-kwalitee ()
"Launch Invenio code kwalitee check on the current buffer."
(interactive)
(let* ((input-file (buffer-file-name (current-buffer)))
(command (concat "~/private/src/invenio-devscripts/invenio-check-kwalitee "
"--check-some "
input-file)))
(save-some-buffers (not compilation-ask-about-save) nil)
(if (and input-file (string-equal (file-name-extension input-file) "py"))
(compilation-start command)
(message "[ERROR] Cannot run Invenio code kwalitee check on non-Python buffers."))))
(global-set-key (kbd "C-c k") 'tibor-invenio-check-kwalitee)This allows you to press C-c k to see the list of potential code
kwalitee problems in your buffer and to press the usual C-x ` to
jump from one problematic location to the next in order to fix them.
If you work on a feature branch named say new-feature that stems from master and you modify plenty of files, add some new files, and delete some other files, then you may want to check how the overall code kwalitee changed in your branch with respect to master. You can run:
invenio-check-branch master new-featureThis devscript will perform kwalitee check on files that were modified in the new-feature branch when compared to the master branch and it will inform you of kwalitee report differences. The comparison disregards line numbers, so that typical code moving/adding/deleting situations are handled relatively nicely, even if the starting branch code is dirty. However, doing comparison in this “fuzzy” way may also leads to false positives, so beware. If/when we are free of kwalitee issues everywhere, we can do stricter comparison here.
You should run invenio-check-branch on your feature branches before
every merge request.
Once you have installed Invenio for the first time on a box, you can
use this devscript to reinstall the Atlantis demo site anew. Please
beware, because invenio-recreate-demo-site will erase your database
tables and recreate your /opt/invenio anew.
Installing Invenio demo site from scratch requires having an Internet
connection and may take up to 15 minutes. For a quicker technique to
restore a vanilla Invenio demo site from a previously installed one,
please see invenio-backup-site and invenio-restore-site devscripts
below.
This devscript launches all unit/regression/web test suite cases on your installation, compares results against the last run, and warns you in case of differences. It is useful to see whether your branch did not accidentally break some tests. The script assumes running on the usual Atlantis demo site conditions, and may be destructive, so please beware.
If you have installed Invenio on your box, e.g. by following
InvenioOnDebian instructions, then you may find it cumbersome to
switch between various branches. Say you are developing a feature
based off the master branch and you would like to quick-fix a bug
occurred in the maint-1.1 branch, as well as to check this behaviour
on the brand new next branch, all without having to recreate your
Invenio demo sites.
The solution is to set up several virtualenv environments, each with
its own installation place and its own database, so that you can
quickly switch between them. The devscript
invenio-initialise-virtualenv will assist you in customising your
environment for Invenio developments.
Some prerequisites, starting out of a system installed in InvenioOnDebian manner:
sudo aptitude install virtualenvwrapper
rm -rf /opt/invenio # this will become symlink laterHere is how you can create a new virtualenv environment called
invenio-master (using system Apache and system Python packages) with
fresh new Invenio demo site on it:
cd ~/private/src/invenio
git checkout master
mkvirtualenv --system-site-packages invenio-master
invenio-initialise-virtualenv invenio-master --yes-i-know
deactivate && workon invenio-master
invenio-recreate-demo-site --yes-i-knowYou can use the above commands to up several virtualenv environments
named invenio-maint-1.0, invenio-maint-1.1, invenio-master,
invenio-next, each corresponding to the respective branch.
Here is how you can quickly switch between them:
workon invenio-maint-1.1 # browser will show we are running 'maint-1.1' site
workon invenio-next # browser will show we are running 'next' siteHere is typical hacking session with switching between environments:
## (1) let's start by working on some-new-feature-a in 'master' branch
workon invenio-master # switch to master virtualenv
git checkout -b some-new-feature-a master # start working on a feature
cd modules/webfoo/lib/
vim webfoo_templates.py # edit some files
invenio-make-install # install changes
firefox # check some-new-feature-a in browser
## (2) phone rings, there is a bug in Invenio v1.1.0 that we have to quick fix
git commit -a -m xxx # stash unsaved work on some-new-feature-a
workon invenio-maint-1.1 # switch to maint-1.1 virtualenv
git checkout -b fix-for-webbar maint-1.1 # start working on a bug fix for WebBar
cd modules/webbar/lib/
vim webbar_dblayer.py # edit some files
invenio-make-install # install changes
firefox # check whether WebBar is OK now
git commit -a -m 'WebBar: fix for baz' # commit the fix
## (3) a visitor enters and wonders about the brand new search facets from 'next' branch
workon invenio-next
firefox # will show 'next' demo site with facetsCreates backup file for the given Invenio site instance. Basically
dumps the current database content and creates tarball of every file
under /opt/invenio. The final backup file is named like
inveniomaint11-site-backup-2013-07-28-13-05-21.tar and can be
restored via invenio-restore-site devscript, see below.
Restores Invenio site instance from the backup file previously created
by invenio-backup-site devscript, see above. Basically, removes
everything under /opt/invenio, and recreates database tables from
the dump.
Creates deployment recipe out of an Invenio commit or a range of commits. Here are typical use cases:
invenio-create-deploy-recipe --cds
invenio-create-deploy-recipe --cds HEAD~10..
invenio-create-deploy-recipe --inspire --via-filecopy 48c7348..52fa18f
invenio-create-deploy-recipe --inspire HEAD,HEADNotes:
- The script understands CDS and INSPIRE site conditions as two
concrete site examples. You can use
--cdsor--inspirecommand line options to specify deployment on either CDS or INSPIRE. - The script takes
SHA1of the commit to deploy, orSHA1a..SHA1bcommit range to deploy. If this argument is missing, then it deploys the latest single commit on the current branch. - Note that the commit range may contain a comma – such as in the
fourth
HEAD,HEADexample above – in which case the commit range before the comma will be taken from the Invenio repository, and the commit range after the comma will be taken from the appropriate overlay repository (CDS or INSPIRE). In case the comma is not specified, the current repository is taken, either Invenio master repository or the INSPIRE overlay. - The generated recipe is
org-mode-formatted and is to be inspected by humans. E.g. in case of DB changes, the recipe will contain a warning at the end of the recipe, and a human is supposed provide appropriateALTER TABLEstatements and the like. - The generated recipe can use either file copy instructions
(
--via-filecopy) or autotools installation instructions (--via-install). You may preferably use the former in order to deploy small patches. - Note that the recipes may be generated on a machine that does not run the same destination overlay. E.g. one can generate Invenio deployment recipes for a remote INSPIRE service on a locally run Atlantis demo site. However, when generated INSPIRE overlay recipes, it is helpful to do so on an INSPIRE site, because the script may try to find location of some expected files.
Once the deployment recipe is proof-read by human, the generated code snippets can be copy-pasted onto appropriate worker nodes.
Note that this devscript constitutes a human-assisted semi-automatic deployment technique. This is mostly because of the bleeding edge nature of the master branch deployment that is seeked out here. If we would like to deploy release maintenance branches, then we could target more fully automated deployment mechanisms, via post-commit hooks or via Fabric.
Checks whether all git committers are well listed in the AUTHORS or THANKS file.
This is used mostly before making releases by people wearing system integration and release management hats.
Checks I18N usage in Invenio. Firstly, checks whether each file from
POTFILES.in exists and whether it contains _(. Secondly, checks
each Invenio source file that contains _( whether it is well present
in POTFILES.in.
This is used mostly before making releases or massive PO file updates by people wearing system integration and release management hats.
A helper script to delete old CVS-era tags from Invenio source code repository. Some developers still have the old tags present in their personal repositories, so when you fetch from them, the old CVS style tags may reappear from time to time. When this happens, running this script will delete them.
Generate and publish Invenio code browser pages. Assumes having
installed Invenio locally first. Call this script with
--generate-code-browser-pages CLI option and check its output in a
web browser. If everything looks fine, then publish generated code
browser pages on the code browser canonical web site by calling this
script with --publish-code-browser-pages CLI option. This script is
used from time to time by the Head Developer.
This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program; if not, see http://www.gnu.org/licenses/.