Playground using mongodb to look at mailman archives
Fetching latest commit…
Cannot retrieve the latest commit at this time
mongomail ========= Playground with mongodb to investigate mailman's archives The goal is to see how mongodb would perform for a (very) large database containing the archives of a mailman. In this case we are using the archives from the fedora-development mailing list. Dependencies: ------------- * pymongo * mongodb-server Install: ------- * Install the dependencies yum install pymongo mongodb-server * Start the server sudo service mongd start * Retrieve the archives from mailman in a specific directory # Currently we retrieve archives from the Fedora project's list: # devel, packaging fr-users mkdir lists cd lists python ../get_mbox.py rm *.gz # remove empty .gz files cd .. * Load the archives into mongodb python to_mongo.py <listname> <path/to/archives> ie: python to_mongo.py devel lists/devel* * See the stats: python stats.py Note: ----- The script to_mongo.py should still be improve to take into account email which are not encoded in UTF-8. When loading the fr-users archives this leads to quite some problems. LICENSE: -------- BSD without advertising: Copyright (c) 2012, Pierre-Yves Chibon <email@example.com> Copyright 1994-2006 The FreeBSD Project. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE FREEBSD PROJECT ''AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FREEBSD PROJECT OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. The views and conclusions contained in the software and documentation are those of the authors and should not be interpreted as representing official policies, either expressed or implied, of the FreeBSD Project.