Bosh bootstrap rate limited

[rkoster]

When running bosh-bootstrap deploy i get:

/Users/rubenkoster/.rvm/gems/ruby-1.9.3-p194@if-bosh/gems/excon-0.20.1/lib/excon/middlewares/expects.rb:10:in `response_call': Expected(200) <=> Actual(413 Request Entity Too Large) (Excon::Errors::RequestEntityTooLarge)
  request => {:chunk_size=>1048576, :connect_timeout=>60, :headers=>{"User-Agent"=>"fog/1.8.0", "Content-Type"=>"application/json", "Accept"=>"application/json", "X-Auth-Token"=>"{token}", "Host"=>"{ip}", "Content-Length"=>91}, :idempotent=>false, :instrumentor_name=>"excon", :middlewares=>[Excon::Middleware::Expects, Excon::Middleware::Idempotent, Excon::Middleware::Instrumentor, Excon::Middleware::Mock], :mock=>false, :nonblock=>true, :read_timeout=>60, :retry_limit=>4, :ssl_ca_file=>"/Users/rubenkoster/.rvm/gems/ruby-1.9.3-p194@if-bosh/gems/excon-0.20.1/data/cacert.pem", :ssl_verify_peer=>true, :uri_parser=>URI, :write_timeout=>60, :host=>"{ip}", :path=>"/compute/v1.1/7276102beb584c66a11bb6b923a4d4f1/os-security-groups.json", :port=>"80", :query=>"ignore_awful_caching1365706883", :scheme=>"http", :user=>nil, :password=>nil, :family=>0, :body=>"{\"security_group\":{\"name\":\"microbosh-openstack-inception-vm\",\"description\":\"inception-vm\"}}", :expects=>200, :method=>"POST", :retries_remaining=>4}
  response => #<Excon::Response:0x007fc7344784b8 @data={:body=>"{\"overLimitFault\": {\"message\": \"This request was rate-limited.\", \"code\": 413, \"details\": \"Only 10 POST request(s) can be made to * every minute.\"}}", :headers=>{"Date"=>"Thu, 11 Apr 2013 19:01:23 GMT", "Content-Type"=>"text/html; charset=UTF-8", "Retry-After"=>"3", "Content-Length"=>"147", "Vary"=>"Accept-Encoding", "Connection"=>"close"}, :status=>413, :remote_ip=>"{ip}"}, @body="{\"overLimitFault\": {\"message\": \"This request was rate-limited.\", \"code\": 413, \"details\": \"Only 10 POST request(s) can be made to * every minute.\"}}", @headers={"Date"=>"Thu, 11 Apr 2013 19:01:23 GMT", "Content-Type"=>"text/html; charset=UTF-8", "Retry-After"=>"3", "Content-Length"=>"147", "Vary"=>"Accept-Encoding", "Connection"=>"close"}, @status=413, @remote_ip="{ip}">

[rkoster]

Similar problem has been solved in bosh cloudfoundry/bosh#99.
The fix is here could maybe reused.

[tpradeep]

Why don't you tackle the problem from OpenStack side. Increase the rate-limits or if you are using devstack simply remove it.

If you want to increase the value, do the following

In your /etc/nova/api-paste.ini find the following

[filter:ratelimit]
paste.filter_factory = nova.api.openstack.compute.limits:RateLimitingMiddleware.factory

Immediately after those lines add the following

limits=(POST, "", ., 9999, MINUTE);(POST, "/servers", ^/servers, 9999, DAY);(PUT, "", ., 9999, MINUTE);(GET, "changes-since", .changes-since., 9999, MINUTE);(DELETE, "", .*, 9999, MINUTE)

Change 9999 to any number and restart nova-api server.

[rkoster]

The problem is I have not control over the openstack environment. It is managed by our hoster.

[tpradeep]

Oops :(

[drnic]

How do we implement this fix via micro_bosh.yml?

[frodenas]

You can't, it should be wrapper for the fog gem, catching the RequestEntityTooLarge exception and retrying.

[drnic]

So either implemented in bosh_openstack_cpi?