Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fog::Storage (Rackspace) directories.get is very slow #2714

Closed
mikhailov opened this issue Feb 26, 2014 · 21 comments

Comments

Projects
None yet
4 participants
@mikhailov
Copy link

commented Feb 26, 2014

We use Rackspace as a backup storage. It's been developed through queue file by file, today with increased load it started to eat too much CPU:

rs_object = Fog::Storage.new(:provider => 'Rackspace'...).directories.get('dir')
rs_object.bucket_obj.files.new(...)
rs_object.save

By some reason it returns redundant details (bytes, count), but all it needs just to upload a file.
We use instantiation each time, how can we cache it? Maybe persistance connection may help?

@elight

This comment has been minimized.

Copy link

commented Feb 26, 2014

@mikhailov Sorry to hear this! I'm looking into it right now.

@mikhailov

This comment has been minimized.

Copy link
Author

commented Feb 26, 2014

thanks, the dependencies:

    excon (0.25.3)
    fog (1.15.0)
      builder
      excon (~> 0.25.0)
      formatador (~> 0.2.0)
      mime-types
      multi_json (~> 1.0)
      net-scp (~> 1.1)
      net-ssh (>= 2.1.3)
      nokogiri (~> 1.5)
      ruby-hmac
    multi_json (1.7.7)

I can't update Gemfile, but applying a patch should be fine.

@elight

This comment has been minimized.

Copy link

commented Feb 26, 2014

@mikhailov Can you set EXCON_DEBUG=1 in your environment and then repeat your test above? This will show you the request and the resulting response.

I don't know that I'm seeing the same results as you are. I went into pry and uploaded a file in a similar fashion to yourself. Here's the response that I saw in the log:

excon.response  {:body=>"", :headers=>{"Last-Modified"=>"Wed, 26 Feb 2014 17:16:08 GMT", "Content-Length"=>"0", "Etag"
=>"794d9d685b517eae2a6b7eba2c2617c9", "Content-Type"=>"text/html; charset=UTF-8", "X-Trans-Id"=>"txa21394f49bed44f3a86
a6-00530e2158dfw1", "Date"=>"Wed, 26 Feb 2014 17:16:08 GMT", "Connection"=>"close"}, :status=>201, :remote_ip=>"174.14
3.184.158"}

Also, could you try updating to latest fog: 1.20.0? I don't know that this will make your problems go away though.

Fog isn't doing too much, really, when you call File#save. It's calling https://github.com/fog/fog/blob/master/lib/fog/rackspace/requests/storage/put_object.rb which sends the HTTP PUT to our API with the right headers via excon (the fog HTTP library).

@mikhailov

This comment has been minimized.

Copy link
Author

commented Feb 26, 2014

Thanks for quick reply! I disabled those lines and the CPU consumption is still high:

rs_object.bucket_obj.files.new(...)
rs_object.save

So my problem is in connection establishment and retrieve certain directory. Can this variable (EXCON_DEBUG=1 ) help to determine why it takes so much CPU?

@mikhailov

This comment has been minimized.

Copy link
Author

commented Feb 26, 2014

I've tested with EXCON_DEBUG=1, it's a bit crazy. excon.response returns tons of files inside the directory! How can I just upload file without retrieving full directory scan?

https://github.com/fog/fog/blob/v1.15.0/lib/fog/rackspace/models/storage/directories.rb#L54-56

I can see the same in master.

@krames

This comment has been minimized.

Copy link
Member

commented Feb 26, 2014

Try this

rs_object = Fog::Storage.new(:provider => 'Rackspace'...).directories.new(:key => 'dir')
rs_object.bucket_obj.files.new(...)
rs_object.save

When you use the get method, it will retrieve the metadata for the first 10,000 objects in the container. Using directories.new(:key => 'dir') will avoid this.

@elight Maybe we need to look into lazily populating the files as this seems to be a pretty common problem.

@mikhailov

This comment has been minimized.

Copy link
Author

commented Feb 26, 2014

@krames you are my saviour! second time, I remember!

@krames

This comment has been minimized.

Copy link
Member

commented Feb 26, 2014

@mikhailov Thanks for the kind words! :)

@mikhailov

This comment has been minimized.

Copy link
Author

commented Feb 26, 2014

I can't believe, one line of code drops the CPU load twice! Application load is the same!
screen shot 2014-02-26 at 20 24 54

@geemus

This comment has been minimized.

Copy link
Member

commented Feb 26, 2014

Thanks!

On Wed, Feb 26, 2014 at 2:26 PM, Anatoly Mikhailov <notifications@github.com

wrote:

I can't believe, one line of code drops the CPU load twice!
[image: screen shot 2014-02-26 at 20 24 54]https://f.cloud.github.com/assets/14601/2275115/29078182-9f24-11e3-82c8-4818de5359f4.png

Reply to this email directly or view it on GitHubhttps://github.com//issues/2714#issuecomment-36172376
.

@krames

This comment has been minimized.

Copy link
Member

commented Feb 27, 2014

Awesome!! 👍

@mikhailov

This comment has been minimized.

Copy link
Author

commented Mar 20, 2014

guys, a related bug again here. It's about network traffic. Please take a look first:
screen shot 2014-03-19 at 18 48 07

Let me explain what happens here. This server handles the media server role: video files are recording and uploading through custom queue application. Queue handles S3 and Rackspace upload both, so I expect outbound/inbound traffice ratio 2/1, but it was 5/1 before I switched Rackspace upload off. Now it's AWS S3 upload only, so you can see 1/1 ratio. The question why Rackspace produces 2-3x traffic?

P.S. S3 upload happens through aws-sdk gem.

@krames

This comment has been minimized.

Copy link
Member

commented Mar 20, 2014

Hmmmm....let's see if we can't get some more debugging information. Can you prefix your command with EXCON_DEBUG=true. This should spit out all of the raw http requests going out to the rackspace cloud. In particular, we are looking at the :path and :method properties. Do you see a lot of certain requests?

@mikhailov

This comment has been minimized.

Copy link
Author

commented Mar 20, 2014

@krames yes, I'll run the debug mode and put the results later... The outbound of traffic seems like O(2^n), where n is income traffic.
screen shot 2014-03-20 at 16 44 09
screen shot 2014-03-20 at 17 11 16

@mikhailov mikhailov reopened this Mar 20, 2014

@krames

This comment has been minimized.

Copy link
Member

commented Mar 20, 2014

Interesting!!! Keep us posted.

@mikhailov

This comment has been minimized.

Copy link
Author

commented Mar 21, 2014

We do use Rackspace in several regions, at the moment I'm doing DEBUG logging analysis and can see extra traffic once file uploads from our server in EU region to Rackspace Sydney (storage101.syd2.clouddrive.com). An overhead is here. I guess it because of retries.

An upload from EU to Rackspace London (storage101.lon3.clouddrive.com) produces expected amount of traffic.

Is there any throttling on Cloud Files?

@krames

This comment has been minimized.

Copy link
Member

commented Mar 21, 2014

There is a rate limit of 100 writes per second. (http://docs.rackspace.com/files/api/v1/cf-devguide/content/Absolute_Limits-d1e942.html) Something tells me that's not the issue you are running into.

Let me see if I can't escalate this to our cloud files team. Can you send me the debugging information for a couple of the failing requests. Specially, I am looking for the path, the X-Trans-Id header, and the data center (which is all Sydney, right?). My email is kyle.rames@rackspace.com.

@mikhailov

This comment has been minimized.

Copy link
Author

commented Mar 21, 2014

@krames I sent you an email. Thanks!

@krames

This comment has been minimized.

Copy link
Member

commented Apr 2, 2014

@mikhailov How are things with your issue?

@krames krames added the rackspace label Apr 2, 2014

@mikhailov

This comment has been minimized.

Copy link
Author

commented Apr 25, 2014

I have disabled that for a while, need to enable and check it out again

@krames

This comment has been minimized.

Copy link
Member

commented Apr 29, 2014

@mikhailov I am going to close this issue, but please feel to re-open it when you get more information.

@krames krames closed this Apr 29, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.