Redmine Xapian search plugin
The current version of Redmine Xapian is 1.6.2
With this plugin you will be able to make searches by file name and by strings inside your documents through the Xapian search engine.
This plugin can also index the files located in your repositories.
This plugin replaces search controller, its view and search methods.
1. Installation and Setup
A copy of the plugin can be downloaded from Github at https://github.com/xelkano/redmine_xapian/downloads
1.1. Required packages
Redmine >= 3.0
To use xapian search engine you must install libxapian-ruby and xapian-omega packages (on Ubuntu). To index some files with omega you may have to install some other packages like xpdf, antiword, …
From Omega documentation:
- PDF (.pdf) if pdftotext is available (comes with xpdf)
- PostScript (.ps, .eps, .ai) if ps2pdf (from ghostscript) and pdftotext (comes with xpdf) are available
- OpenOffice/StarOffice documents (.sxc, .stc, .sxd, .std, .sxi, .sti, .sxm, .sxw, .sxg, .stw) if unzip is available
- OpenDocument format documents (.odt, .ods, .odp, .odg, .odc, .odf, .odb, .odi, .odm, .ott, .ots, .otp, .otg, .otc, .otf, .oti, .oth) if unzip is available
- MS Word documents (.doc, .dot) if antiword is available
- MS Excel documents (.xls, .xlb, .xlt) if xls2csv is available (comes with catdoc)
- MS Powerpoint documents (.ppt, .pps) if catppt is available (comes with catdoc)
- MS Office 2007 documents (.docx, .dotx, .xlsx, .xlst, .pptx, .potx, .ppsx) if unzip is available
- Wordperfect documents (.wpd) if wpd2text is available (comes with libwpd)
- MS Works documents (.wps, .wpt) if wps2text is available (comes with libwps)
- AbiWord documents (.abw)
- Compressed AbiWord documents (.zabw) if gzip is available
- Rich Text Format documents (.rtf) if unrtf is available
- Perl POD documentation (.pl, .pm, .pod) if pod2text is available
- TeX DVI files (.dvi) if catdvi is available
- DjVu files (.djv, .djvu) if djvutxt is available
- XPS files (.xps) if unzip is available
On debian you can use this command to install required indexing tools:
# apt-get install unzip antiword ghostscript catdoc xpdf libwpd-tools libwps-tools gzip unrtf
Since Redmine 1.4, Redmine uses Bundler and required xapian gem can be installed running bundle command in redmine root directory. To install required xapian gem you might need to install uuid-dev package.
On debian run:
# apt-get install uuid-dev
1.2. Plugin installation
Install redmine_xapian into the plugins directory with:
# cd into [redmine_root] # bundle install --without development test # bundle exec rake redmine:plugins:migrate RAILS_ENV=production
Now, go to the Redmine root directory and run bundle command to install required gems:
# cd /var/www/redmine # bundle
And after that restart the application server.
Now, you can see new check boxes “files” and “repositories” on search screen, those allows you to search attachments by file name and its contents.
Xapian plugin checks for ruby bindings before its startup, if the plugin can not find them, it is not activated and this message is going to appear in Redmine log “REDMAIN_XAPIAN ERROR: No Ruby bindings for Xapian installed !!. PLEASE install Xapian search engine interface for Ruby" If you see this message, please make sure that Xapian library are correctly installed for your ruby environment.
First at all, go to redmine interface and in plugins section configure the plugin. It’s very important to set up correctly the directory that will contain the Xapian databases.
To use stemmed languages in searches, Xapian needs to generate stemmed forms for each language and store them in a database.
Before you can use the plugin, you have to populate the Xapian databases for each stemmed language you want to use. From plugin version 1.6 a new script (xapian_indexer.rb) is included in the extra directory that is use to populate Xapian databases. This script is a wrapper for omindex binary, so omindex should not to be used anymore.
First edit the script file and configure the required variables, they are self-explanatory and most of them can be specified and overwritten through the command line.
To view command line help simply run “xapian_index.rb -h”. To view the complete process you can run it using verbose flag (-v).
Note, that the first time you index a repository it can take a long time. After the first indexation the process is much faster.
You have to take into account that the script needs a defined repository identifier for all repositories that will be indexed. The repository identifier is not required in redmine and can have null value for some of the repositories, so before indexing a repository you have to be sure that the repository has a defined identifier, otherwise the script will ignore it.
You can run this MySql statement in your database for setting up a default identifier for your repositories:
mysql> update repositories set identifier='main' where identifier is null;
Examples running the script:
Run the script based in its configuration in verbose mode, index repositories and files:
# ruby xapian_indexer.rb -v
Only index redmine files:
# ruby xapian_indexer.rb -v -f
Only index repositories:
# xapian_indexer.rb -v -r
Indexing repositories of some projects:
# xapian_indexer.rb -v -r -p project_identifier1,project_identifier2
Once you have tested the script functionality you can configure it for running every day at nights keeping your xapian database up to date, for example:
00 * * * * /usr/bin/xapian_indexer.rb
After file indexing and the plugin setup on settings you can search attachment by name and content.
To perform searches in a particular module such as Files, Documents,… it is
mandatory to enable these modules in your projects and to have corresponding
permissions view_*; Be sure it is checked.