Bug: [fog][WARNING] Unrecognized arguments: region, use_iam_profile

[mattheworiordan]

For the last 3-6 months we occasionally get an error from fog using IAM authentication on an EC2 box as follows:

[fog][WARNING] Unrecognized arguments: region, use_iam_profile

The odd thing is we built in a retry mechanism to work around this but it made no difference because it seems like once the Ruby environment is setup, if the problem is there, it will remain there until you exit the Ruby process. Typically running it again resolves the problem.

We've have the same problem authenticating locally too i.e. from a local machine connecting to Amazon EC2 with the credentials in the ~/.fog file.

We get:

[fog][WARNING] Unrecognized arguments: region
undefined method `servers' for #<Fog::Compute::AWS::Real:0x007fd811307a78> (NoMethodError)

I assumed this was only me being affected as no one else reported the issue, but recently other users in the team have been reporting the problem as well.

I have tried to resolve this myself locally for months now and got nowhere with this :(

Any help or advice would be greatly appreciated.

[geemus]

Strange. Are you doing any multi-thread stuff? I ask as it looks a bit like autoloading-weirdness, which I think can occur in that situation.

[mattheworiordan]

Strange. Are you doing any multi-thread stuff? I ask as it looks a bit like autoloading-weirdness, which I think can occur in that situation.

I don't think those errors occur when we're doing "multi-thread stuff", but that's quite plausible, we use https://github.com/grosser/parallel quite extensively to speed up our queries by hitting all regions simultaneously. Can you think of any sensible way to address that i.e. can I force the loading of types, initialize the library before using threads etc? Any ideas appreciated.

[geemus]

Yeah, that might do it (I understand autoload can break in multi-thread cases). Explicitly requiring things would prevent the issue, though it might be a bit annoying it could probably be meta-programmed to work. Otherwise, I suppose if you make the first call in the mail thread, before fanning out, it should ensure it has everything loaded (this is of course annoying in it's own right).

[mattheworiordan]

Thanks @geemus. I am away on holiday right now so won't make progress on this until next week. I thought about those approaches the other day in fact. I may even simply add a pause of an arbitrary 1-2 seconds on all subsequent threads to ensure that autoload threading issues are avoided. I'll play around with some ideas next week and confirm which one is best for anyone else suffering the same problem.

Thanks again

[geemus]

Sounds good. Wish I could more explicitly help, but I have only heard about this second hand. I think it's the same kind of risk auto-load always imposes (rather than something particular to fog), so perhaps there are other precedents we could draw from too. Anyway, enjoy the holiday and let me know if there is anything else I can help with when you return. Thanks!

[mattheworiordan]

@geemus I have deployed a fix and this appears to be working so far, although it was always intermittent. I'll reopen this issue if it turns out that the issue is not to do with autoloading.

If anyone is interested, my rather simplistic solution to resolve the problem of autoloading and parallel threads was to change our Fog methods such as:

def ec2(region)
  Fog::Compute::AWS.new(region: region, use_iam_profile: true)
end

which failed intermittently with:

Parallel.each(regions) do |region|
   ec2(region).do_something
end

into this:

def ec2(region)
  parallel_safe(:ec2) do
    Fog::Compute::AWS.new(region: region, use_iam_profile: true)
  end
end

def parallel_safe(type, &block)
  @parallel_safe_timestamp ||= {}

  if !@parallel_safe_timestamp[type]
    @parallel_safe_timestamp[type] = (Time.now + 0.5).to_f
    yield
  else
    safe_wait_time = @parallel_safe_timestamp[type] - Time.now.to_f
    sleep safe_wait_time if safe_wait_time > 0
    yield
  end
end

[geemus]

@mattheworiordan Hope that nails it down for you and thanks for taking the time to share your solution.