Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Playground using mongodb to look at mailman archives

branch: master

Fetching latest commit…

Octocat-spinner-32-eaf2f5

Cannot retrieve the latest commit at this time

Octocat-spinner-32 .gitignore
Octocat-spinner-32 README
Octocat-spinner-32 TODO
Octocat-spinner-32 get_mbox.py
Octocat-spinner-32 stats.py
Octocat-spinner-32 thread.py
Octocat-spinner-32 to_mongo.py
README
mongomail
=========

Playground with mongodb to investigate mailman's archives

The goal is to see how mongodb would perform for a (very) large database containing the archives of a mailman.

In this case we are using the archives from the fedora-development mailing list.



Dependencies:
-------------

* pymongo
* mongodb-server


Install:
-------

* Install the dependencies
yum install pymongo mongodb-server

* Start the server
sudo service mongd start

* Retrieve the archives from mailman in a specific directory
# Currently we retrieve archives from the Fedora project's list:
# devel, packaging fr-users

mkdir lists
cd lists
python ../get_mbox.py
rm *.gz  # remove empty .gz files
cd ..

* Load the archives into mongodb
python to_mongo.py <listname> <path/to/archives>
 ie: python to_mongo.py devel lists/devel*

* See the stats:
python stats.py


Note:
-----
The script to_mongo.py should still be improve to take into account
email which are not encoded in UTF-8.
When loading the fr-users archives this leads to quite some problems.


LICENSE: 
--------


BSD without advertising:

Copyright (c) 2012, Pierre-Yves Chibon <pingou@pingoured.fr>

Copyright 1994-2006 The FreeBSD Project. All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE FREEBSD PROJECT ''AS IS'' AND ANY EXPRESS OR
IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
EVENT SHALL THE FREEBSD PROJECT OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

The views and conclusions contained in the software and documentation are those
of the authors and should not be interpreted as representing official policies,
either expressed or implied, of the FreeBSD Project.

Something went wrong with that request. Please try again.