process_in_background option seems to ignore versions when process_<model>_upload=true

[bantic]

I am attempting to test using carrierwave_backgrounder asynchronously, and every time I run it on the rails console, my thumbnail version does not get generated.

I'm storing the files on s3, and using sidekiq as my backgrounder queue.

Snippet of my photo model (photo.rb), which uses a field of the same name (photo) to store the carrierwave file:

...
mount_uploader :photo, PhotoUploader
process_in_background :photo
...

photo_uploader.rb:

...
storage :fog
include ::CarrierWave::Backgrounder::Delay
include CarrierWave::MiniMagick
process :resize_to_fit => [1200,900]
version :thumb do
  process :resize_to_fill => [200,150]
end

When I do the following on the rails console and then look at my s3 bucket, I only see one version, no thumbnail version:

>> photo_model = Photo.new
>> photo_model.remote_photo_url='http://media-cache-ec3.pinterest.com/192/a1/f1/6c/a1f16c832b0653b839f03547aa8787da.jpg' # random pinterest photo
>> photo_model.process_photo_upload = true
>> photo_model.save!
# no thumbnail version on s3

If I do not set process_photo_upload to true and I run sidekiq in another terminal tab, the thumbnail does get generated properly.

Is there something I'm doing wrong?

[lardawge]

Couple things,

What version of carrierwave and CB are you using?

Have you tried passing it a local file?

[bantic]

Here are the versions of carrierwave_* gems I'm using:

  * carrierwave (0.8.0)
  * carrierwave_backgrounder (0.2.0)
  * carrierwave_direct (0.0.8)

And yes, the same issue is reproduced with a local file, e.g.:

>>> p = Photo.new
>>> p.photo = File.open('./img.jpg')
>>> p.process_photo_upload = true
>>> p.save!

=> no thumbnail version on s3

[bantic]

To make sure carrierwave_direct wasn't conflicting things somehow, I removed it from my gemfile (and rebundled) and tried the above again, with the same result.

[lardawge]

Had a chance to look at this...

I found a bug related to Rmagick but no issue with versions... Not sure where to go with this one. Strange... Can you reproduce in a clean app?

[lardawge]

I am also wondering if you're introducing a bug by having the model and attachment with the same name... Just a thought.

[bantic]

I tried changing the column names from photo to picture and photo_process to picture_processing -- same issue remains, unfortunately.
If I can narrow it down any further, I will post an update here.

[hyperrjas]

I have the same problem, I can not see the versions created on my s3 bucket. I can see only the original file. I'm using delayed_job.

This is my uploader:

class AvatarUploader < CarrierWave::Uploader::Base
  include CarrierWave::MiniMagick
  include ::CarrierWave::Backgrounder::Delay
  include Sprockets::Helpers::RailsHelper
  include Sprockets::Helpers::IsolatedHelper
  storage :fog
  #storage :grid_fs

  def store_dir
    "uploads/#{model.class.to_s.underscore}/#{mounted_as}/#{model.id}"
  end

  version :thumb do
    process :resize_to_limit => [50,50]
  end

  version :big do
    process :resize_to_limit => [180, nil]
  end                                

  def extension_white_list
    %w(jpg jpeg gif png)
  end

 def filename
   random_token = Digest::SHA2.hexdigest("#{Time.now.utc}--#{model.id.to_s}").first(20)
   ivar = "@#{mounted_as}_secure_token"    
   token = model.instance_variable_get(ivar)
   token ||= model.instance_variable_set(ivar, random_token)
   "#{model.id}_#{token}.jpg" if original_filename
  end
end

In my user.rb model

mount_uploader :avatar, AvatarUploader, mount_on: :avatar_filename
process_in_background :avatar

Please, how can I add versions with this marvelous gem. My versions is working fine with carrierwave. I'm using carrierwave (0.8.0).

Thank you very much!

[lardawge]

@hyperrjas, are you starting up sidekiq correctly? It may not be processing the job. You need to specify the queue you set in the CB config when you start sidekiq or the jobs won't get processed. I am sure there is a closed ticket addressing this. There really should be a reminder in the documentation.

[lardawge]

Like this:

#config/initializers/carrierwave_backgrounder.rb
CarrierWave::Backgrounder.configure do |c|
  c.backend :sidekiq, queue: :bg_photos
end

Start sidekiq with the following;

sidekiq -c 3 -q bg_photos -q default

@bantic and @hyperrjas
Is that what you're doing?

[bantic]

@lardawge I was using it with carrierwave's default queue (called carrierwave, I think?). I remember seeing that the queue was processing items. And in my case, using the background method (via sidekiq) did correctly generate the versions and upload to s3. My issue was that when I set #process_<model>_upload=true (to do it synchronously), the versions failed to generate.

[hyperrjas]

@lardawge I'm using delayed_job.

#config/initializers/carrierwave_backgrounder.rb
CarrierWave::Backgrounder.configure do |c|
  c.backend :delayed_job, queue: :carrierwave
  # c.backend :resque, queue: :carrierwave
  # c.backend :sidekiq, queue: :carrierwave
  # c.backend :girl_friday, queue: :carrierwave
  # c.backend :qu, queue: :carrierwave
  # c.backend :qc
end

Is required use sidekiq to create versions??

Thank you very much!

[lardawge]

@bantic, An interesting discovery... if I set photo_model.process_photo_upload = true before I set the remote image url, all works as expected. So this is a confirmed bug.

For now the workaround is to do things in the order below

>> photo_model = Photo.new
>> photo_model.process_photo_upload = true
>> photo_model.remote_photo_url='http://media-cache-ec3.pinterest.com/192/a1/f1/6c/a1f16c832b0653b839f03547aa8787da.jpg' # random pinterest photo
>> photo_model.save!

@hyperrjas, can you confirm this works or are you seeing another issue?

[hyperrjas]

@lardawge thank you for your response but I am not using sidekiq gem. I'm using delayed_job. I'm using mongodb as database with mongoid. I have not installed redis on my server and I can not install redis.

#config/initializers/carrierwave_backgrounder.rb
CarrierWave::Backgrounder.configure do |c|
  c.backend :delayed_job, queue: :carrierwave
  # c.backend :resque, queue: :carrierwave
  # c.backend :sidekiq, queue: :carrierwave
  # c.backend :girl_friday, queue: :carrierwave
  # c.backend :qu, queue: :carrierwave
  # c.backend :qc
end

Is it neccessary use sidekiq to create versions? or is it possible create versions with delayed_job?

Thank you very much!

[lardawge]

@hyperrjas, This issue is regarding overriding carrierwave_backgrounder with process_<model>_upload= not working correctly. If you're having issues setting up with delayed job, please open an issue and post everything you did including how you are starting delayed_job to process the background jobs. It sounds to me like you are not actually processing jobs if you are not seeing versions getting created. I personally have two apps using delayed job and have no issues.

[lukebergen]

Not sure if it's related but I noticed that you're using the "mount_on" option in user.rb.

I'm trying to work through an issue myself where specifying this option in the model causes carrierwave_backgrounder to not work correctly (for me at least).

@bantic As a test, if you pull the mount_on: :avatar_filename option out of user.rb (and make sure the straight-up "avatar" field is in the DB for CW) does it start working? It did for me and now I'm just trying to figure out how to move forward since I need to keep the mount_on option in.

[hyperrjas]

@lardawge Can you paste an app example with delayed_job working and mongoid?

@lukebergen I don't understand you mean. How is working your model with delelayed_job?. In https://github.com/jnicklas/carrierwave-mongoid there is a example with mount_on and for me is working fine without carrierwave_backgrounder:

class Dokument
  mount_uploader :upload, DokumentUploader, mount_on: :upload_filename
end

Can you paste a example that is working with mongoid, delayed_job and carrierwave_backgrounder?

Thank you!

[lukebergen]

@hyperrjas Sorry, I was confusing your code with @bantic's. I'm using sidekiq, not delayed_job.

I was just saying that for me, if I pull out the "mount_on" option from the "mount_uploader" call, carrierwave_background starts working for me.

My code is something like this:

app/models/attachment.rb

class Attachment
  include Mongoid::Document
  # the code as it is now fails to actually queue the sidekiq job.
  # but if I comment out the ', mount_on: "file_filename"' part, it works.
  mount_uploader :file, FileUploader, mount_on: "file_filename"
  process_in_background :file
  ...
end

app/uploaders/file_uploader.rb

class FileUploader < CarrierWave::Uploader::Base
  include CarrierWave::Backgrounder::Delay
  include CarrierWave::MiniMagick
  version :"s50x50" do
    process resize_and_pad: [50, 50]
  end
end

[lardawge]

@lukebergen, What is not working correctly when you use mounted_on?

[lukebergen]

@lardawge, It just never seems to send the job off to sidekiq. I'm digging into it now to try to get something more specific than that.

[lardawge]

Try passing your mounted on name as the arg to process_in_background

class Attachment
  include Mongoid::Document
  # the code as it is now fails to actually queue the sidekiq job.
  # but if I comment out the ', mount_on: "file_filename"' part, it works.
  mount_uploader :file, FileUploader, mount_on: "file_filename"
  process_in_background :file_filename
  ...
end

[lukebergen]

Yeah, that was one of the first things I tried.

When I do that and run the following to test:

att = Attachment.new
att.file = File.open("some_big_file.jpg")
att.save

The line "att.file = File.open" seems to cause the processing to happen right there, it's like not specifying ":file" causes backgrounder to not link up with carrierwave and carrierwave goes and does it's own processing.

Still digging.

Edit: Also, when I do the 3 lines above with the model having "process_in_background :file_filename" the line "att.save" gives me a stacktrace with error "undefined method remove_file_filename?' for #<Attachment:0x007ff3b8ceb978>" originating at carrierwave_backgrounder-0.2.1/lib/backgrounder/orm/base.rb:103:inenqueue_file_filename_background_job?'

[lardawge]

I figured that was a giant leap... can you open up a issue on this and add some details so we don't clutter this one. They aren't related as far as I can tell.

[lukebergen]

Yup, created here: #130

If I figure it out I'll submit a PR