Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Doesn't call :on_success #8

Open
ngw opened this Issue Jun 15, 2011 · 4 comments

Comments

Projects
None yet
3 participants

ngw commented Jun 15, 2011

Hi, I need to upload some files on S3, and I would like to cleanup the tmp structure I created before on_success.

def upload_file dir, file
  f = dir + '/' + file
  on_error = Proc.new { |http| puts "An error occured: #{http.response_header.status}" }
  cleanup = Proc.new { |http| puts "\nDELETE!\n"; FileUtils.rm( f ) }
  item = Happening::S3::Item.new(
    @config[ 'portfolios_bucket' ], @subdomain + "/" + mime_dir( file ) + file,
    :aws_access_key_id => @config[ 's3_access_key' ], 
    :aws_secret_access_key => @config[ 's3_secret_key' ],
    :permissions => 'public-read' )
  item.put( File.read( f ), :on_error => on_error, :on_success => cleanup,
    :headers => { "Content-Type" => File.mime_type?( f ) } ) do |response|
  end
end

Files are correctly uploaded, but I never see "DELETE". The app is built using Goliath.io, I'm not entirely sure how it integrates with EM, but considering that it's pretty big and everything seems to work very well ... Should I call EM.stop?
Suggestions on how to fix?

Collaborator

jweiss commented Jun 16, 2011

Are you only not seeing the puts or does the cleanup also not happen?
I do not know where Goliath is outputting SDTOUT.

EM.stop is not needed and would cause problems with Goliath

ngw commented Jun 16, 2011

Honestly I'm a bit confused now, I don't understand the 'meaning' of on_success ...

def upload_file dir, file
  f = dir + '/' + file
  on_error = Proc.new { |http| puts "An error occured: #{http.response_header.status}" }
  item = Happening::S3::Item.new(
    @config[ 'portfolios_bucket' ], @subdomain + "/" + mime_dir( file ) + file,
    :aws_access_key_id => @config[ 's3_access_key' ], 
    :aws_secret_access_key => @config[ 's3_secret_key' ],
    :permissions => 'public-read' )
  item.put( File.read( f ), :on_error => on_error, :on_success => cleanup( f ),
    :headers => { "Content-Type" => File.mime_type?( f ) } ) do |response|
    puts "UPLOADED: #{f}"
  end
end

def cleanup file
  unless file =~ /assets/
    FileUtils.rm file
    puts "RM #{ file }"
  end    
end

Output is:

RM /Users/ngw/kenji/tmp/nofeed/css/main.css
RM /Users/ngw/kenji/tmp/nofeed/css/reset.css
RM /Users/ngw/kenji/tmp/nofeed/index.html
RM /Users/ngw/kenji/tmp/nofeed/project.zip
[46807:INFO] 2011-06-16 14:36:32 :: Status: 200, Content-Length: 2, Response Time: 279.25ms
UPLOADED: /Users/ngw/kenji/tmp/nofeed/css/main.css
UPLOADED: /Users/ngw/kenji/tmp/nofeed/css/reset.css
UPLOADED: /Users/ngw/kenji/assets/mediator/mediator.html
UPLOADED: /Users/ngw/kenji/tmp/nofeed/index.html
UPLOADED: /Users/ngw/kenji/tmp/nofeed/project.zip

The files aren't removed at all ....
When is on_success called?

ngw commented Jun 16, 2011

Damn, and that was the result of an uncleaned dir.
So, the code is suboptimal, I still need to refactor it, but I will include the full source.
It downloads something from s3, process it, and upload the results on s3. The problem is that I'm apparently bitten by concurrency problems, all related to happening and on_success.

https://gist.github.com/79abf5219627d4db814e

What happens basically is that it starts processing (a processing that is entirely external) before it downloaded all files, apparently tries to cleanup before it finished uploading. I'm not able to understand if the problem is in Goliath or Happening, and I'm definitely not sure how to fix this ....

leoc commented Oct 7, 2011

Is this still an issue?

I'd suggest not giving a block to the put function, because that overrides the :on_success option.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment