Issue reading from /tmp directory in build environment

[kbaum]

I am getting the following error when running tests from concurrent builds.

/usr/bin/ruby18 -I "/usr/lib64/ruby/gems/1.8/gems/cucumber-0.3.96/lib:lib" "/usr/lib64/ruby/gems/1.8/gems/cucumber-0.3.96/bin/cucumber" --profile firefox35 --format pretty --format html --out=/data/homedirs/cosmos/.hudson/jobs/firefox35/workspace/log/cucumber/html/cucumber_firefox35_results.html features/current
Using the firefox35 profile...
Unable to load thrift_native extension. Defaulting to pure Ruby libraries.
spawned dj pid 4556

• User {:username=>"demo1"}
  No such file or directory - /tmp/mms2r/9F8813E4DB0BB740A4E475B593C6E4D31E38928Fgsmbnop12esfirmwidecorpgscom/2/1253664813.98513.html (Errno::ENOENT)
  /usr/lib64/ruby/gems/1.8/gems/mms2r-2.2.0/lib/mms2r/media.rb:418:in initialize' /usr/lib64/ruby/gems/1.8/gems/mms2r-2.2.0/lib/mms2r/media.rb:418:inopen'
  /usr/lib64/ruby/gems/1.8/gems/mms2r-2.2.0/lib/mms2r/media.rb:418:in process_media' /usr/lib64/ruby/gems/1.8/gems/mms2r-2.2.0/lib/mms2r/media.rb:354:inprocess'
  (eval):3:in each_without_optional_block' (eval):3:ineach'
  /usr/lib64/ruby/gems/1.8/gems/mms2r-2.2.0/lib/mms2r/media.rb:351:in process' /usr/lib64/ruby/gems/1.8/gems/mms2r-2.2.0/lib/mms2r/media.rb:233:ininitialize'
  /usr/lib64/ruby/gems/1.8/gems/mms2r-2.2.0/lib/mms2r/media.rb:113:in orig_new' /usr/lib64/ruby/gems/1.8/gems/mms2r-2.2.0/lib/mms2r/media.rb:113:innew'
  /data/homedirs/cosmos/.hudson/jobs/firefox35/workspace/app/models/email/message.rb:113:in rfc822_attachments' /data/homedirs/cosmos/.hudson/jobs/firefox35/workspace/app/models/email/message.rb:99:inbuild_from_tmail'
  ./db/populate/02_demo_users.rb:21

It looks to me that this is because more than one process is reading the same email with the same message_id concurrently.

Looking at the code, all processes running as the same user share the same tmp directory.

def self.tmp_dir
  @@tmp_dir ||= File.join(Dir.tmpdir, (ENV['USER'].nil? ? '':ENV['USER']), 'mms2r')
end

And the media_dir is a product of the tmpdir using hte message id and the current time:

  @media_dir = File.join(self.tmp_dir(), 
                 self.safe_message_id(@mail.message_id))

[monde]

Could you be re-using the same mail fixtures over and over in your test suite? If so, the test suite could be clobbering results. If that is the case you could submit a patch that creates a truely random director for @media_dir

[kbaum]

This could indeed be the case.

For generating random file names, I've had success using:

UUIDTools::UUID.random_create.to_s

Do you think that would make sense?

thx

[monde]

I think you can keep it simple by introducing the current time or a random number as part of the directory hierarchy. Then messages that are processed multiple times will be retained in unique directory hierarchies

@media_dir = File.join(self.tmp_dir(),
                     Time.now.to_i.to_s,
                     self.safe_message_id(@mail.message_id))

or

@media_dir = File.join(self.tmp_dir(),
                     (rand(1e7)).to_s,
                     self.safe_message_id(@mail.message_id))

[kbaum]

I don't like using time since that could break in a multithreaded environment. I'm not sure of the specifics of the rand function but i know uuid is guaranteed to be unique.

  @media_dir = File.join(self.tmp_dir(),
                 "#{self.safe_message_id(@mail.message_id)}_#{UUIDTools::UUID.random_create}")

[monde]

I'm hesitant to bring in a dependency on the uuidtools gem (http://uuidtools.rubyforge.org/) into the main functionality of MMS2R to satisfy an somewhat edge case scenario in a testing environment.

[kbaum]

Makes sense. I would use an alternate method of generating a random unique string if I new it would always work. I have ran into issues in the past with rand methods that were based on time and not always that random.

I'm not too sure that you would never run into a scenario like this in multi-threaded environments. To me it just seems like we are making mms2r that much more bullet proof.

thx.

[monde]

In a threaded environment, given an app like Twitpic that posts MMS media to a feed, if the app starts processing the same message multiple times, then that's a flaw with the application, not with MMS2R. In a scenario like that, the app could work on each message using a work queue, or other similar methodology. MMS2R makes the assumption that each message is unique in terms of the value in its Message-ID header as can be seen in the logic to derive a temporary directory to decode all of its media.

To me, in a threaded testing environment that has no work queue, and is setting up and tearing down the same message fixture over and over, that 1:1,000,000 randomness is good enough. Such as:

@media_dir = File.join(self.tmp_dir(),
                     (rand(1e7)).to_s,
                     self.safe_message_id(@mail.message_id))

[kbaum]

Makes sense.

[monde]

I added your fork into the latest 2.4.0 version if you hadn't noticed, thanks