Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Import many messages from multiple accounts #14

Merged
merged 24 commits into from

3 participants

@tomafro

As discussed yesterday, these are the changes I made to enable sauron to both import a lot of messages from different accounts and show them quickly through the web interface.

tomafro added some commits
@tomafro tomafro Split searching for message uids from loading the messages.
This will allow more flexible message loading strategies, such as loading messages in batches or only loading specific messages.
2a44d4c
@tomafro tomafro Select the INBOX when initialising the Gmail client
As we never operate on any other mailboxes, there's no need to select a mailbox before each operation.  This makes the client specific to the single mailbox, which given the mailbox-centric way that IMAP works seems to be a useful property.
ca81eda
@tomafro tomafro Sometimes it's nice to fetch a single message given a UID 101af51
@tomafro tomafro Store messages in the repository against a given key, using the messa…
…ge UID

This will allow us to check whether the repository already contains the given key, allowing us in the future to only import messages that aren't yet in the repository.
a9d3348
@tomafro tomafro Don't reimport messages already in the repository e7e4be6
@tomafro tomafro Move MD5 key generation right down to the message store.
If we're storing messages in the repository with a given id, key generation is redundant as far as the repository goes.  It's now the responsibility of the message store itself to decide how to save files.  For now we're still using MD5 hashes but this could easily change.
49f79ad
@tomafro tomafro Import ALL the email from a user's account, not just their inbox mess…
…ages

I've been testing against my tom.ward account and sauron-test, neither of which have enough messages to really break anything (in their inboxes at least).  Switching to All Mail to try and destroy things.
8c9bf02
@tomafro tomafro Use Mail to bypass encoding issues
Attempting to import all the emails from tom.ward@gofreerange.com I encountered some nasty encoding problems like this one:

  Encoding::UndefinedConversionError: "\xA3" from ASCII-8BIT to UTF-8
    from /Users/tomw/Projects/freerange/sauron/lib/file_based_message_store.rb:14:in `write'
    from /Users/tomw/Projects/freerange/sauron/lib/file_based_message_store.rb:14:in `[]='

Rather than worry too much about this, I decided to pipe all imported messages through Mail.new to see if that fixed the problem.  It seemed to, but we should be aware of encoding problems that may arise and fix the underlying problem when or if it becomes the next most important thing to do.
26cfc09
@tomafro tomafro Use Base64 encoding when storing messages to avoid encoding issues.
Passing messages through Mail.new(message).to_s worked for about 1,100 of my messages, but not all of them.  Base64 is safer.
28c7357
@lazyatom

I'm pleased that you hit this issue already :)

@lazyatom

I know this isn't a concern at this point, but we know that UID isn't going to be unique across multiple accounts (or even mailboxes).

tomafro added some commits
@tomafro tomafro GmailImapClient => GoogleMail::Mailbox
What was previously the client only provides access to a single mailbox, so can be called Mailbox with little confusion.  This makes many things clearer to me, such as referring to 'all the messages in the mailbox' (rather than the client).
de48666
@tomafro tomafro The GoogleMail::Mailbox#messages method might be useful one day, but …
…is not currently used
05305cc
@tomafro tomafro Give the FileBasedMessageStore a default root path b1499b4
@tomafro tomafro Change MessageRepository to use a FileBasedMessageStore by default 06443e1
@tomafro tomafro Let's add messages to the repository, not store them. af73267
@tomafro tomafro Only the MessageRepository itself should need to call #instance 3fff3e8
@tomafro tomafro Cucumber should use AccountMessageImporter to import an account's mes…
…sages
0029d6b
@tomafro tomafro MessageRespository#includes? clashes with Class#includes?. Use #exist…
…s? instead.
7fc571e
@tomafro tomafro Return our own Message object from the repository, not a Mail object 609d067
@tomafro tomafro Introduced a CachedConnection, storing all cached values in an IMAP c…
…ache
6dacbae
@tomafro tomafro Use an ActiveRecord model to record each imported message.
To display a list of messages, there's no need to store them as files on disk.  Only their subject, date and from is required, so store these as records in the database.

One advantage of storing the messages on disk is that it avoids having to reimport data (which can be very slow).  However, using the CachedConnection to IMAP should negate this.
50752c3
@tomafro tomafro Support importing messages from multiple accounts.
As the initial schema hasn't been merged into master, I didn't see any problem adding a column to the migration directly, rather than cluttering up the world with more migrations.
fcd4995
@tomafro tomafro Add the ability to show a full message.
In this case, full takes its most extreme meaning - the entire content of the original email.  This is more as proof that we can still access original message content via the web than anything else.

In order to prevent the display of a list of messages taking ages, the original message is only loaded at the point it is needed, not when returning one or a number of messages from the repository.
27de04a
@lazyatom

I think it would be really helpful to explain your thinking behind the introduction of this.

Sure. I got bored importing messages again and again and again because it took so long. Previously (and maybe still at this commit) there was a message store which placed each message on disk, but if I wanted to change the import process (adding a database record for example), or change the message store itself, there was no easy way to get at a whole bunch of messages without re-downloading them. Fine for 10s of messages, but crap when you're dealing with more than a hundred, say.

I'd noticed that imap calls were split into two parts, one to find the ids of the messages you wanted, and one to download the actual data. It occurred to me that if I cached the data calls (uid_fetch) as low down as possible, I could try out different importers over a large range of data much more easily. I was also interested in how easy it would be to change the connection class. In the end it was very simple (though I'm sure the tests could be improved).

@lazyatom lazyatom merged commit 324ec35 into master
@floehopper

I looked for a migration called CreateMessages, which is what I'd expect by convention, but didn't find it. What's the thinking behind calling this InitialSchema?

Not put too much thinking into it, but it's something I've done on a lot of recent personal projects. I find that at the start of a project the schema can be in a state of flux. Until the data on production can't be recreated I put everything in a single migration, moving things around and changing them at will. I only add more migrations once I actually need to migrate production (not just rake db:migrate:reset).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Mar 23, 2012
  1. @tomafro

    Split searching for message uids from loading the messages.

    tomafro authored
    This will allow more flexible message loading strategies, such as loading messages in batches or only loading specific messages.
  2. @tomafro

    Select the INBOX when initialising the Gmail client

    tomafro authored
    As we never operate on any other mailboxes, there's no need to select a mailbox before each operation.  This makes the client specific to the single mailbox, which given the mailbox-centric way that IMAP works seems to be a useful property.
  3. @tomafro
  4. @tomafro

    Store messages in the repository against a given key, using the messa…

    tomafro authored
    …ge UID
    
    This will allow us to check whether the repository already contains the given key, allowing us in the future to only import messages that aren't yet in the repository.
  5. @tomafro
  6. @tomafro

    Move MD5 key generation right down to the message store.

    tomafro authored
    If we're storing messages in the repository with a given id, key generation is redundant as far as the repository goes.  It's now the responsibility of the message store itself to decide how to save files.  For now we're still using MD5 hashes but this could easily change.
  7. @tomafro

    Import ALL the email from a user's account, not just their inbox mess…

    tomafro authored
    …ages
    
    I've been testing against my tom.ward account and sauron-test, neither of which have enough messages to really break anything (in their inboxes at least).  Switching to All Mail to try and destroy things.
  8. @tomafro

    Use Mail to bypass encoding issues

    tomafro authored
    Attempting to import all the emails from tom.ward@gofreerange.com I encountered some nasty encoding problems like this one:
    
      Encoding::UndefinedConversionError: "\xA3" from ASCII-8BIT to UTF-8
        from /Users/tomw/Projects/freerange/sauron/lib/file_based_message_store.rb:14:in `write'
        from /Users/tomw/Projects/freerange/sauron/lib/file_based_message_store.rb:14:in `[]='
    
    Rather than worry too much about this, I decided to pipe all imported messages through Mail.new to see if that fixed the problem.  It seemed to, but we should be aware of encoding problems that may arise and fix the underlying problem when or if it becomes the next most important thing to do.
  9. @tomafro

    Use Base64 encoding when storing messages to avoid encoding issues.

    tomafro authored
    Passing messages through Mail.new(message).to_s worked for about 1,100 of my messages, but not all of them.  Base64 is safer.
Commits on Mar 26, 2012
  1. @tomafro

    GmailImapClient => GoogleMail::Mailbox

    tomafro authored
    What was previously the client only provides access to a single mailbox, so can be called Mailbox with little confusion.  This makes many things clearer to me, such as referring to 'all the messages in the mailbox' (rather than the client).
  2. @tomafro
Commits on Mar 27, 2012
  1. @tomafro
  2. @tomafro
  3. @tomafro
  4. @tomafro
  5. @tomafro
  6. @tomafro
  7. @tomafro
  8. @tomafro
  9. @tomafro

    Use an ActiveRecord model to record each imported message.

    tomafro authored
    To display a list of messages, there's no need to store them as files on disk.  Only their subject, date and from is required, so store these as records in the database.
    
    One advantage of storing the messages on disk is that it avoids having to reimport data (which can be very slow).  However, using the CachedConnection to IMAP should negate this.
  10. @tomafro

    Support importing messages from multiple accounts.

    tomafro authored
    As the initial schema hasn't been merged into master, I didn't see any problem adding a column to the migration directly, rather than cluttering up the world with more migrations.
  11. @tomafro

    Add the ability to show a full message.

    tomafro authored
    In this case, full takes its most extreme meaning - the entire content of the original email.  This is more as proof that we can still access original message content via the web than anything else.
    
    In order to prevent the display of a list of messages taking ages, the original message is only loaded at the point it is needed, not when returning one or a number of messages from the repository.
  12. @tomafro

    Travis CI needs the migrations to be run before tests will pass. It d…

    tomafro authored
    …oes not do this automatically.
Commits on Mar 29, 2012
  1. @lazyatom
This page is out of date. Refresh to see the latest.
Showing with 454 additions and 164 deletions.
  1. +3 −1 .travis.yml
  2. +4 −0 app/controllers/messages_controller.rb
  3. +2 −2 app/views/messages/index.html.erb
  4. +8 −0 app/views/messages/show.html.erb
  5. +3 −3 config/database.yml
  6. +2 −2 config/environments/test.rb
  7. +1 −0  config/routes.rb
  8. +11 −0 db/migrate/20120327123609_initial_schema.rb
  9. +9 −1 db/schema.rb
  10. +2 −2 features/step_definitions/walking_skeleton.rb
  11. +2 −3 lib/account_message_importer.rb
  12. +25 −0 lib/cache_backed_message_store.rb
  13. +22 −6 lib/file_based_message_store.rb
  14. +0 −33 lib/gmail_imap_client.rb
  15. +21 −0 lib/google_mail/imap_cache.rb
  16. +61 −0 lib/google_mail/mailbox.rb
  17. +7 −5 lib/message_importer.rb
  18. +49 −14 lib/message_repository.rb
  19. +12 −2 test/fakes/fake_gmail.rb
  20. +8 −1 test/functional/messages_controller_test.rb
  21. +6 −7 test/unit/account_message_importer_test.rb
  22. +32 −9 test/unit/file_based_message_store_test.rb
  23. +0 −38 test/unit/gmail_imap_client_test.rb
  24. +13 −0 test/unit/google_mail/imap_cache_test.rb
  25. +93 −0 test/unit/google_mail/mailbox_test.rb
  26. +19 −5 test/unit/message_importer_test.rb
  27. +39 −30 test/unit/message_repository_test.rb
View
4 .travis.yml
@@ -1,2 +1,4 @@
rvm:
- - 1.9.3
+ - 1.9.3
+before_script:
+ - "bundle exec rake db:migrate db:test:prepare"
View
4 app/controllers/messages_controller.rb
@@ -4,4 +4,8 @@ class MessagesController < ApplicationController
def index
@messages = MessageRepository.messages
end
+
+ def show
+ @message = MessageRepository.find(params[:id])
+ end
end
View
4 app/views/messages/index.html.erb
@@ -4,8 +4,8 @@
<% @messages.each do |message| %>
<li class="message">
<span class="date"><%= message.date.strftime("%Y-%m-%D %H:%M:%S") %></span>
- <span class="subject"><%= message.subject %></span>
- <span class="sender"><%= message.from.first %></span>
+ <span class="subject"><%= link_to message.subject, message_path(message) %></span>
+ <span class="sender"><%= message.from %></span>
</li>
<% end %>
</ul>
View
8 app/views/messages/show.html.erb
@@ -0,0 +1,8 @@
+<ul>
+<li><%= @message.subject %></li>
+<li><%= @message.date %></li>
+<li><%= @message.from %></li>
+</ul>
+<pre>
+<%= @message.original.to_s %>
+</pre>
View
6 config/database.yml
@@ -5,7 +5,7 @@
# gem 'sqlite3'
development:
adapter: sqlite3
- database: ":memory:"
+ database: "db/development.sqlite3"
pool: 5
timeout: 5000
@@ -14,13 +14,13 @@ development:
# Do not set this db to the same as development or production.
test: &test
adapter: sqlite3
- database: ":memory:"
+ database: "db/test.sqlite3"
pool: 5
timeout: 5000
production:
adapter: sqlite3
- database: ":memory:"
+ database: "db/production.sqlite3"
pool: 5
timeout: 5000
View
4 config/environments/test.rb
@@ -37,5 +37,5 @@
end
require Rails.root + 'test' + 'fakes' + 'fake_gmail'
-require 'gmail_imap_client'
-GmailImapClient.connection_class = FakeGmail::Connection
+require 'google_mail/mailbox'
+GoogleMail::Mailbox.connection_class = FakeGmail::Connection
View
1  config/routes.rb
@@ -1,5 +1,6 @@
Sauron::Application.routes.draw do
root to: "messages#index"
+ resources :messages, only: [:show]
if Rails.env.test?
# So that we can test arbitrary test controllers but avoid exposing this catch-all route in production
View
11 db/migrate/20120327123609_initial_schema.rb
@@ -0,0 +1,11 @@
+class InitialSchema < ActiveRecord::Migration
+ def change
+ create_table :messages do |table|
+ table.string :account
+ table.string :uid
+ table.string :subject
+ table.datetime :date
+ table.string :from
+ end
+ end
+end
View
10 db/schema.rb
@@ -11,6 +11,14 @@
#
# It's strongly recommended to check this file into your version control system.
-ActiveRecord::Schema.define(:version => 0) do
+ActiveRecord::Schema.define(:version => 20120327123609) do
+
+ create_table "messages", :force => true do |t|
+ t.string "account"
+ t.string "uid"
+ t.string "subject"
+ t.datetime "date"
+ t.string "from"
+ end
end
View
4 features/step_definitions/walking_skeleton.rb
@@ -9,12 +9,12 @@
Mail.new("Subject: Message one\nDate: 2012-05-23 12:34:45\nFrom: Dave"),
Mail.new("Subject: Message two\nDate: 2012-06-22 09:21:31\nFrom: Barry")
].each do |message|
- FakeGmail.server.accounts[account].add_message('INBOX', message)
+ FakeGmail.server.accounts[account].add_message(message)
end
end
When /^the messages for account "([^"]*)" are imported$/ do |account|
- MessageImporter.new(GmailImapClient.connect(account, 'password')).import_into(MessageRepository.instance)
+ AccountMessageImporter.import_for(account, 'password')
end
Then /^they should be visible on the messages page$/ do
View
5 lib/account_message_importer.rb
@@ -1,11 +1,10 @@
-require 'gmail_imap_client'
require 'message_repository'
class AccountMessageImporter
class << self
def import_for(email, password)
- imap_client = GmailImapClient.connect(email, password)
- MessageImporter.new(imap_client).import_into(MessageRepository.instance)
+ mailbox = GoogleMail::Mailbox.connect(email, password)
+ MessageImporter.new(mailbox).import_into(MessageRepository)
end
end
end
View
25 lib/cache_backed_message_store.rb
@@ -0,0 +1,25 @@
+class CacheBackedMessageStore
+ def initialize(cache = ActiveSupport::Cache::FileStore.new(Rails.root + 'data' + Rails.env + 'messages'))
+ @cache = cache
+ end
+
+ def add(account, uid, message)
+ @cache.write [account, uid], message
+ end
+
+ def find(account, uid)
+ @cache.read [account, uid]
+ end
+
+ class << self
+ delegate :add, :find, to: :instance
+
+ def instance
+ @instance ||= new
+ end
+
+ def configure(*args)
+ @instance = new(*args)
+ end
+ end
+end
View
28 lib/file_based_message_store.rb
@@ -1,19 +1,35 @@
require 'fileutils'
+require 'base64'
class FileBasedMessageStore
- def initialize(root_path)
+ attr_reader :root_path
+
+ def initialize(root_path = Rails.root + 'data' + Rails.env + 'messages')
@root_path = root_path
end
+ def include?(key)
+ File.exist? key_path(key)
+ end
+
+ def [](key)
+ if include?(key)
+ Base64.strict_decode64(File.read(key_path(key)))
+ end
+ end
+
def []=(key, value)
- full_path = File.expand_path(key, @root_path)
- FileUtils.mkdir_p File.dirname(full_path)
- File.write full_path, value
+ FileUtils.mkdir_p File.dirname(key_path(key))
+ File.write key_path(key), Base64.strict_encode64(value)
end
def values
- Dir["#{@root_path}/**"].map do |path|
- File.read(path)
+ Dir["#{root_path}/**"].map do |path|
+ Base64.strict_decode64(File.read(path))
end
end
+
+ def key_path(key)
+ File.expand_path(Digest::MD5.hexdigest(key.to_s), root_path)
+ end
end
View
33 lib/gmail_imap_client.rb
@@ -1,33 +0,0 @@
-require 'net/imap'
-
-class GmailImapClient
- class AuthenticatedConnection
- delegate :examine, :uid_search, :uid_fetch, to: :@imap
-
- def initialize(email, password)
- @imap = ::Net::IMAP.new 'imap.gmail.com', 993, true
- @imap.login email, password
- end
- end
-
- cattr_accessor :connection_class
- self.connection_class = AuthenticatedConnection
-
- attr_reader :connection
-
- def initialize(connection)
- @connection = connection
- end
-
- def inbox_messages
- connection.examine 'INBOX'
- uids = connection.uid_search('ALL')
- connection.uid_fetch(uids, 'BODY.PEEK[]').map {|m| m.attr['BODY[]']}
- end
-
- class << self
- def connect(email, password)
- new connection_class.new(email, password)
- end
- end
-end
View
21 lib/google_mail/imap_cache.rb
@@ -0,0 +1,21 @@
+module GoogleMail
+ class ImapCache
+ delegate :read, :write, :fetch, to: :@cache
+
+ def initialize(cache = ActiveSupport::Cache::FileStore.new(Rails.root + 'tmp' + 'cache' + 'imap'))
+ @cache = cache
+ end
+
+ class << self
+ delegate :read, :write, :fetch, to: :instance
+
+ def instance
+ @instance ||= new
+ end
+
+ def configure(*args)
+ @instance = new(*args)
+ end
+ end
+ end
+end
View
61 lib/google_mail/mailbox.rb
@@ -0,0 +1,61 @@
+require 'net/imap'
+
+module GoogleMail
+ class Mailbox
+ class AuthenticatedConnection
+ attr_reader :email
+ delegate :examine, :uid_search, :list, :uid_fetch, to: :@imap
+
+ def initialize(email, password)
+ @imap = ::Net::IMAP.new 'imap.gmail.com', 993, true
+ @imap.login email, password
+ @email = email
+ end
+ end
+
+ class CachedConnection
+ delegate :email, :examine, :uid_search, :list, to: :@connection
+
+ def initialize(email, password, cache = GoogleMail::ImapCache)
+ @connection = AuthenticatedConnection.new(email, password)
+ @cache = cache
+ end
+
+ def uid_fetch(uid, command)
+ @cache.fetch [@connection.email, uid, command] do
+ @connection.uid_fetch(uid, command)
+ end
+ end
+ end
+
+ cattr_accessor :connection_class
+ self.connection_class = CachedConnection
+
+ attr_reader :connection
+ delegate :email, to: :connection
+
+ def initialize(connection)
+ @connection = connection
+ mailboxes = @connection.list('', '%').collect(&:name)
+ if mailboxes.include?('[Gmail]')
+ @connection.examine '[Gmail]/All Mail'
+ else
+ @connection.examine '[Google Mail]/All Mail'
+ end
+ end
+
+ def uids
+ connection.uid_search('ALL')
+ end
+
+ def message(uid)
+ connection.uid_fetch(uid, 'BODY.PEEK[]').map {|m| m.attr['BODY[]']}.first
+ end
+
+ class << self
+ def connect(email, password)
+ new connection_class.new(email, password)
+ end
+ end
+ end
+end
View
12 lib/message_importer.rb
@@ -1,13 +1,15 @@
class MessageImporter
- attr_reader :message_client
+ attr_reader :mailbox
- def initialize(message_client)
- @message_client = message_client
+ def initialize(mailbox)
+ @mailbox = mailbox
end
def import_into(repository)
- message_client.inbox_messages.each do |message|
- repository.store(message)
+ mailbox.uids.each do |uid|
+ unless repository.exists?(mailbox.email, uid)
+ repository.add mailbox.email, uid, mailbox.message(uid)
+ end
end
end
end
View
63 lib/message_repository.rb
@@ -1,37 +1,72 @@
require 'mail'
class MessageRepository
- class KeyGenerator
- def key_for(message)
- Digest::MD5.hexdigest(message)
+ class Record < ActiveRecord::Base
+ self.table_name = :messages
+ end
+
+ class Message
+ attr_reader :record, :original
+ delegate :subject, :date, :from, :to_param, to: :record
+
+ def initialize(record, original = "")
+ @record = record
+ @original = original
+ end
+
+ def ==(message)
+ message.is_a?(Message) &&
+ message.record == record
+ end
+ end
+
+ class LazyOriginalMessage
+ def initialize(account, uid, store)
+ @account = account
+ @uid = uid
+ @store = store
+ end
+
+ def to_s
+ @message ||= @store.find(@account, @uid)
end
end
class << self
attr_writer :instance
- delegate :messages, to: :instance
+ delegate :find, :add, :exists?, :messages, to: :instance
def instance
- @instance ||= new(FileBasedMessageStore.new("data/#{Rails.env}"))
+ @instance ||= new
end
end
- delegate :key_for, to: :@key_generator
- attr_reader :message_store
+ attr_reader :model
+
+ def initialize(model = Record, store = CacheBackedMessageStore)
+ @model = model
+ @store = store
+ end
+
+ def add(account, uid, message)
+ mail = Mail.new(message)
+ @model.create! account: account, uid: uid, subject: mail.subject, date: mail.date, from: mail.from.first
+ @store.add account, uid, message
+ end
- def initialize(store, key_generator = KeyGenerator.new)
- @message_store = store
- @key_generator = key_generator
+ def exists?(account, uid)
+ @model.where(account: account, uid: uid).exists?
end
- def store(message)
- message_store[key_for(message)] = message
+ def find(id)
+ record = @model.where(id: id).first
+ record && Message.new(record, LazyOriginalMessage.new(record.account, record.uid, @store))
end
def messages
- message_store.values.map do |message|
- Mail.new message
+ @model.all.map do |record|
+ Message.new record, LazyOriginalMessage.new(record.account, record.uid, @store)
end
end
end
View
14 test/fakes/fake_gmail.rb
@@ -3,8 +3,9 @@
module FakeGmail
class Server
class Account
- def add_message(mailbox, message)
- mailboxes[mailbox] << message.object_id
+ def add_message(message)
+ mailboxes['[Gmail]']
+ mailboxes['[Gmail]/All Mail'] << message.object_id
messages[message.object_id] = message.to_s
end
@@ -30,6 +31,8 @@ def accounts
self.server ||= Server.new
class Connection
+ attr_reader :email
+
def initialize(email, password)
@email = email
@password = password
@@ -39,6 +42,13 @@ def examine(mailbox)
@mailbox = mailbox
end
+ def list(box, search)
+ unless box == '' && search == '%'
+ raise 'Mock only supports list with no chosen box and a % search'
+ end
+ account.mailboxes.keys.map { |name| Net::IMAP::MailboxList.new([], '/', name) }
+ end
+
def uid_search(name)
raise 'Mock only supports ALL' unless name == 'ALL'
account.mailboxes[@mailbox]
View
9 test/functional/messages_controller_test.rb
@@ -6,10 +6,17 @@ class MessagesControllerTest < ActionController::TestCase
@request.env["HTTP_AUTHORIZATION"] = "Basic " + Base64::encode64("admin:password")
end
- test "#index assigns messages from Gmail" do
+ test "#index finds messages via repository" do
messages = [Mail.new("FROM: George\nDate: 2012-01-01 12:00:00"), Mail.new("FROM: Bob\nDate: 2012-01-01 12:00:00")]
MessageRepository.stubs(:messages).with().returns(messages)
get :index
assert_equal messages, assigns[:messages]
end
+
+ test "#show finds message via repository" do
+ message = stub(subject: 'a', from: 'b', date: Time.now, original: 'Whut')
+ MessageRepository.stubs(:find).with('1234').returns(message)
+ get :show, id: '1234'
+ assert_equal message, assigns[:message]
+ end
end
View
13 test/unit/account_message_importer_test.rb
@@ -3,28 +3,27 @@
class AccountMessageImporterTest < ActiveSupport::TestCase
test 'establishes an imap connection using the given credentials' do
- GmailImapClient.expects(:connect).with('dave@example.com', 'password')
+ GoogleMail::Mailbox.expects(:connect).with('dave@example.com', 'password')
MessageImporter.stubs(:new).returns(stub_everything)
AccountMessageImporter.import_for('dave@example.com', 'password')
end
test 'uses the established connection for importing' do
- gmail_client = stub('gmail-client')
- GmailImapClient.stubs(:connect).returns(gmail_client)
- MessageImporter.expects(:new).with(gmail_client).returns(stub_everything)
+ mailbox = stub('mailbox')
+ GoogleMail::Mailbox.stubs(:connect).returns(mailbox)
+ MessageImporter.expects(:new).with(mailbox).returns(stub_everything)
AccountMessageImporter.import_for('whatever', 'whatever')
end
test 'uses the default message repository' do
- GmailImapClient.stubs(:connect).returns(stub('gmail-client'))
+ GoogleMail::Mailbox.stubs(:connect).returns(stub('mailbox'))
message_repository = stub('message-repository')
importer = stub('importer')
- MessageRepository.stubs(:instance).returns(message_repository)
MessageImporter.stubs(:new).returns(importer)
- importer.expects(:import_into).with(message_repository)
+ importer.expects(:import_into).with(MessageRepository)
AccountMessageImporter.import_for('whatever', 'whatever')
end
View
41 test/unit/file_based_message_store_test.rb
@@ -8,23 +8,46 @@ class FileBasedMessageStoreTest < ActiveSupport::TestCase
FileUtils.mkdir_p TEST_ROOT_PATH
end
- test 'stores messages in given root path' do
- store = FileBasedMessageStore.new(TEST_ROOT_PATH)
- store['x'] = 'y'
- assert_equal 'y', File.read(File.expand_path('x', TEST_ROOT_PATH))
+ test 'uses "data/<Rails.env>/messages" as its default root path' do
+ assert_equal Rails.root + 'data' + 'test' + 'messages', FileBasedMessageStore.new.root_path
+ end
+
+ test 'determines path to store keys using root path and MD5 hash of key' do
+ hash = Digest::MD5.hexdigest('message-key')
+ FileBasedMessageStore.new(TEST_ROOT_PATH)['message-key'] = 'something'
+ end
+
+ test 'stores messages persistently on the filesystem' do
+ FileBasedMessageStore.new(TEST_ROOT_PATH)['x'] = 'y'
+ assert_equal 'y', FileBasedMessageStore.new(TEST_ROOT_PATH)['x']
+ FileUtils.rm_rf TEST_ROOT_PATH
+ assert_nil FileBasedMessageStore.new(TEST_ROOT_PATH)['x']
end
- test 'stores messages successfully when directory doesn\'t exist beforehand' do
+ test 'creates storage directory if it doesn\'t already exist' do
store = FileBasedMessageStore.new(TEST_ROOT_PATH)
FileUtils.rm_rf 'tmp/test'
store['x'] = 'y'
- assert File.directory?('tmp/test/data')
+ assert File.directory?(TEST_ROOT_PATH)
+ end
+
+ test 'encodes messages to avoid problems with strange encodings' do
+ store = FileBasedMessageStore.new(TEST_ROOT_PATH)
+ store['strange'] = "\xA3"
+ assert_equal "\xA3", store['strange']
+ end
+
+ test 'indicates if a key has already been stored' do
+ store = FileBasedMessageStore.new(TEST_ROOT_PATH)
+ refute store.include?('a')
+ store['a'] = 'b'
+ assert store.include?('a')
end
- test 'retrieves all messages from its root path' do
+ test 'provides access to all messages stored' do
store = FileBasedMessageStore.new(TEST_ROOT_PATH)
- File.write(File.expand_path('a', TEST_ROOT_PATH), '1')
- File.write(File.expand_path('b', TEST_ROOT_PATH), '2')
+ store['a'] = '1'
+ store['b'] = '2'
assert_equal ['1', '2'], store.values.sort
end
View
38 test/unit/gmail_imap_client_test.rb
@@ -1,38 +0,0 @@
-require 'test_helper'
-
-class GmailImapClient::AuthenticatedConnectionTest < ActiveSupport::TestCase
- test "should connect to the gmail imap server" do
- Net::IMAP.expects(:new).with('imap.gmail.com', 993, true).returns(stub_everything)
- GmailImapClient::AuthenticatedConnection.new('email', 'password')
- end
-
- test "should login using the supplied email and password" do
- email, password = "email", "password"
- imap = stub("imap")
- imap.expects(:login).with(email, password)
- Net::IMAP.stubs(:new).returns(imap)
- GmailImapClient::AuthenticatedConnection.new(email, password)
- end
-end
-
-class GmailImapClientTest < ActiveSupport::TestCase
- test "should return a new client with connection created with supplied credentials" do
- client = stub('client')
- connection = stub('connection')
- GmailImapClient.connection_class.stubs(:new).with('email', 'password').returns(connection)
- GmailImapClient.stubs(:new).with(connection).returns(client)
- assert_equal client, GmailImapClient.connect('email', 'password')
- end
-
- test "should retrieve all inbox messages from the 'INBOX'" do
- connection = stub("imap-connection")
- connection.expects(:examine).with("INBOX")
- connection.stubs(:uid_search).with("ALL").returns(["uid-1", "uid-2"])
- connection.stubs(:uid_fetch).with(["uid-1", "uid-2"], "BODY.PEEK[]").returns([
- stub(attr: {"BODY[]" => "raw-message-body-1"}),
- stub(attr: {"BODY[]" => "raw-message-body-2"})
- ])
- client = GmailImapClient.new(connection)
- client.inbox_messages
- end
-end
View
13 test/unit/google_mail/imap_cache_test.rb
@@ -0,0 +1,13 @@
+require 'test_helper'
+
+module GoogleMail
+ class ImapCacheTest < ActiveSupport::TestCase
+ test 'delegates read, write and fetch to the underlying cache' do
+ underlying_cache = stub('cache', read: :read_result, write: :write_result, fetch: :fetch_result)
+ cache = ImapCache.new(underlying_cache)
+ assert_equal :read_result, cache.read
+ assert_equal :write_result, cache.write
+ assert_equal :fetch_result, cache.fetch
+ end
+ end
+end
View
93 test/unit/google_mail/mailbox_test.rb
@@ -0,0 +1,93 @@
+require 'test_helper'
+
+module GoogleMail
+ class Mailbox::AuthenticatedConnectionTest < ActiveSupport::TestCase
+ test "should connect to the gmail imap server" do
+ Net::IMAP.expects(:new).with('imap.gmail.com', 993, true).returns(stub_everything)
+ Mailbox::AuthenticatedConnection.new('email', 'password')
+ end
+
+ test "should login using the supplied email and password" do
+ email, password = "email", "password"
+ imap = stub("imap")
+ imap.expects(:login).with(email, password)
+ Net::IMAP.stubs(:new).returns(imap)
+ Mailbox::AuthenticatedConnection.new(email, password)
+ end
+
+ test "gives access to the email address the account represents" do
+ email, password = "email", "password"
+ imap = stub("imap", login: nil)
+ Net::IMAP.stubs(:new).returns(imap)
+ assert_equal 'email', Mailbox::AuthenticatedConnection.new(email, password).email
+ end
+ end
+
+ class Mailbox::CachedConnectionTest < ActiveSupport::TestCase
+ test "instantiates an AuthenticatedConnection to make imap requests through" do
+ cache = stub('cache')
+ Mailbox::AuthenticatedConnection.expects(:new).with('email', 'password')
+ Mailbox::CachedConnection.new('email', 'password', cache)
+ end
+
+ test "delegates imap requests through its connection" do
+ cache = stub('cache')
+ connection = stub('connection')
+ Mailbox::AuthenticatedConnection.stubs(:new).returns(connection)
+ cached_connection = Mailbox::CachedConnection.new('email', 'password', cache)
+ connection.expects(:uid_search).with('ALL')
+ cached_connection.uid_search 'ALL'
+ end
+
+ test "uses cache to avoid repeated calls to uid_fetch" do
+ cache = ActiveSupport::Cache::MemoryStore.new
+ connection = stub('connection', email: 'tom@example.com')
+ Mailbox::AuthenticatedConnection.stubs(:new).returns(connection)
+ cached_connection = Mailbox::CachedConnection.new('email', 'password', cache)
+ connection.stubs(:uid_fetch).with([1, 2, 3], 'ALL').returns(:result)
+ assert_equal :result, cached_connection.uid_fetch([1, 2, 3], 'ALL')
+ connection.stubs(:uid_fetch).never
+ assert_equal :result, cached_connection.uid_fetch([1, 2, 3], 'ALL')
+ end
+ end
+
+ class MailboxTest < ActiveSupport::TestCase
+ test "should return a new mailbox with connection created with supplied credentials" do
+ mailbox = stub('mailbox')
+ connection = stub('connection')
+ Mailbox.connection_class.stubs(:new).with('email', 'password').returns(connection)
+ Mailbox.stubs(:new).with(connection).returns(mailbox)
+ assert_equal mailbox, Mailbox.connect('email', 'password')
+ end
+
+ test "selects [Gmail]/All Mail mailbox if it exists" do
+ connection = stub('connection')
+ connection.stubs(:list).with('', '%').returns([stub(name: 'Anything'), stub(name: '[Gmail]')])
+ connection.expects(:examine).with('[Gmail]/All Mail')
+ Mailbox.new(connection)
+ end
+
+ test "selects [Google Mail]/All Mail mailbox if there is no [Gmail] mailbox" do
+ connection = stub('connection')
+ connection.stubs(:list).with('', '%').returns([stub(name: 'Anything')])
+ connection.expects(:examine).with('[Google Mail]/All Mail')
+ Mailbox.new(connection)
+ end
+
+ test "returns uids of all messages in the mailbox" do
+ connection = stub('imap-connection', examine: nil, list: [])
+ connection.stubs(:uid_search).with('ALL').returns [1, 2, 3, 4]
+ mailbox = Mailbox.new(connection)
+ assert_equal [1, 2, 3, 4], mailbox.uids
+ end
+
+ test "returns a single message given its uid" do
+ connection = stub('imap-connection', examine: nil, list: [])
+ connection.stubs(:uid_fetch).with(1, 'BODY.PEEK[]').returns [
+ stub(attr: {"BODY[]" => "raw-message-body-1"})
+ ]
+ mailbox = Mailbox.new(connection)
+ assert_equal 'raw-message-body-1', mailbox.message(1)
+ end
+ end
+end
View
24 test/unit/message_importer_test.rb
@@ -1,12 +1,26 @@
require 'test_helper'
class MessageImporterTest < ActiveSupport::TestCase
- test 'imports messages' do
- gmail_client = stub('gmail-client', inbox_messages: [:message1, :message2])
- importer = MessageImporter.new(gmail_client)
+ test 'imports messages available in the mailbox' do
+ mailbox = stub('mailbox', email: 'tom@example.com')
+ mailbox.stubs(:uids).returns([3, 4])
+ mailbox.stubs(:message).with(3).returns(:message1)
+ mailbox.stubs(:message).with(4).returns(:message2)
+ importer = MessageImporter.new(mailbox)
+ repository = stub('repository', exists?: false)
+ repository.expects(:add).with('tom@example.com', 3, :message1)
+ repository.expects(:add).with('tom@example.com', 4, :message2)
+ importer.import_into(repository)
+ end
+
+ test 'skips messages already available in repository' do
+ mailbox = stub('mailbox', email: 'tom@example.com')
+ mailbox.stubs(:uids).returns([5])
+ mailbox.expects(:message).with(5).never
+ importer = MessageImporter.new(mailbox)
repository = stub('repository')
- repository.expects(:store).with(:message1)
- repository.expects(:store).with(:message2)
+ repository.stubs(:exists?).with('tom@example.com', 5).returns(true)
+ repository.expects(:add).never
importer.import_into(repository)
end
end
View
69 test/unit/message_repository_test.rb
@@ -1,43 +1,52 @@
require 'test_helper'
class MessageRepositoryTest < ActiveSupport::TestCase
- test 'delegates key generation to a MessageRepository::KeyGenerator by default' do
- key_generator = stub('key-generator')
- MessageRepository::KeyGenerator.expects(:new).returns(key_generator)
- repository = MessageRepository.new(stub_everything)
- key_generator.stubs(:key_for).with(:message).returns(:standard_generated_key)
- assert_equal :standard_generated_key, repository.key_for(:message)
+ test 'uses MessageRepository::Record by default' do
+ model = stub('model')
+ assert_equal MessageRepository::Record, MessageRepository.new.model
end
- test 'allows different key generators to be injected' do
- key_generator = stub('key-generator')
- repository = MessageRepository.new(stub_everything, key_generator)
- key_generator.stubs(:key_for).with(:message).returns(:custom_key)
- assert_equal :custom_key, repository.key_for(:message)
+ test 'adds messages by creating a model' do
+ model = stub('model')
+ message = Mail.new(subject: 'Subject', from: 'tom@example.com', date: Date.today).to_s
+ repository = MessageRepository.new(model)
+ model.expects(:create!).with(account: 'sam@example.com', uid: 123, subject: 'Subject', from: 'tom@example.com', date: Date.today)
+ repository.add('sam@example.com', 123, message)
end
- test 'stores messages in message store' do
- store = stub('message-store')
- repository = MessageRepository.new(store)
+ test 'adds original message data to message store' do
+ model = stub('model', create!: nil)
+ store = stub('store')
+ message = Mail.new(subject: 'Subject', from: 'tom@example.com', date: Date.today).to_s
+ repository = MessageRepository.new(model, store)
+ store.expects(:add).with('sam@example.com', 123, message)
+ repository.add('sam@example.com', 123, message)
+ end
- message = stub('message')
- repository.stubs(:key_for).with(message).returns(:a_message_key)
- store.expects(:[]=).with(:a_message_key, message)
- repository.store(message)
+ test 'uses model to check if messages already exist' do
+ model = stub('model')
+ repository = MessageRepository.new(model)
+ scope = stub('scope')
+ model.stubs(:where).with(account: 'sam@example.com', uid: 1).returns(scope)
+ scope.stubs(:exists?).returns(true)
+ assert repository.exists?('sam@example.com', 1)
end
- test 'retrieves all messages from the message store' do
- store = stub('message-store')
- repository = MessageRepository.new(store)
- store.stubs(:values).returns(["Subject: One", "Subject: Two"])
- assert_equal [Mail.new("Subject: One"), Mail.new("Subject: Two")], repository.messages
+ test 'retrieves all messages from the model' do
+ model = stub('model')
+ repository = MessageRepository.new(model)
+ message = stub('message', account: 'tom@example.com', uid: 123)
+ model.stubs(:all).returns([message])
+ assert_equal [MessageRepository::Message.new(message)], repository.messages
end
-end
-class MessageRepository::KeyGeneratorTest < ActiveSupport::TestCase
- test 'generates keys by calculating MD5 hash of the message' do
- generator = MessageRepository::KeyGenerator.new
- hash = Digest::MD5.hexdigest('a-message')
- assert_equal hash, generator.key_for('a-message')
+ test 'finds a single message from the model' do
+ model = stub('model')
+ scope = stub('scope')
+ repository = MessageRepository.new(model)
+ message = stub('message', account: 'tom@example.com', uid: 123)
+ model.stubs(:where).with(id: '123').returns(scope)
+ scope.stubs(:first).returns(message)
+ assert_equal MessageRepository::Message.new(message), repository.find('123')
end
-end
+end
Something went wrong with that request. Please try again.