DataSourceOpenStack: retry also when first waiting

[david-caro]

Proposed Commit Message

There's a first step to wait and filter out non responsive metadata
services, and the retries option is not taken into account there.
This patch adds the retrying logic to that first step too.

summary: DataSourceOpenStack: retry also when first waiting

There's a first step to wait and filter out non responsive metadata
services, and the `retries` option is not taken into account there.
This patch adds the retrying logic to that first step too.

LP: #1979049

Test Steps

Checklist:

• My code follows the process laid out in the documentation
• I have updated or added any unit tests accordingly
• I have updated or added any documentation accordingly

This depends on #1527 (CLA signing)

[andrewbogott]

Yes please! This would help with overloaded metadata servers and could also help if we're rotating through a set of servers some of which are unresponsive.

[TheRealFalcon]

Depends on #1527

[TheRealFalcon]

@david-caro , We should already be retrying until we hit the max_wait. See my inline comment. Does that sound right to you?

On some level, I think that if the datasource isn't ready within 10 seconds, that's really an issue that should be fixed on the datasource side, but I understand why longer timeouts can help. Historically, we've been hesitant to increase the default timeouts in openstack because on certain architectures that could lead to longer timeouts for non-openstack datasources. See #501 for context.

That said, those concerns aren't as relevant anymore as most other datasources do get reported correctly during early boot.

Can the issues you're seeing be fixed by updating the configuration for your particular openstack installation? Configuration can be set in /etc/cloud/cloud.cfg, /etc/cloud/cloud.cfg.d, or as vendordata to all of your instances.

[david-caro]

@david-caro , We should already be retrying until we hit the max_wait. See my inline comment. Does that sound right to you?

Hmm, I think I understand the logic now, tell me if I'm right:

So when doing the wait_for_metadata_service, the helper function it uses, url_helper.wait_for_url, will try the urls passed, potentially one by one, with the timeout given for each connection, and until a maximum elapsed time of max_wait_seconds, so in order to make it retry the urls that are passed, expecting each of them to timeout in timeout_seconds, should be max_wait_seconds > timeout_seconds * num_urls, is that so?

If so, I can try tweaking the configuration yes,

On some level, I think that if the datasource isn't ready within 10 seconds, that's really an issue that should be fixed on the datasource side, but I understand why longer timeouts can help. Historically, we've been hesitant to increase the default timeouts in openstack because on certain architectures that could lead to longer timeouts for non-openstack datasources. See #501 for context.

I agree with that, but we are trying to minimize the impact until we fix it properly.

That said, those concerns aren't as relevant anymore as most other datasources do get reported correctly during early boot.

Can the issues you're seeing be fixed by updating the configuration for your particular openstack installation? Configuration can be set in /etc/cloud/cloud.cfg, /etc/cloud/cloud.cfg.d, or as vendordata to all of your instances.

[TheRealFalcon]

so in order to make it retry the urls that are passed, expecting each of them to timeout in timeout_seconds, should be max_wait_seconds > timeout_seconds * num_urls, is that so?

Yes, but note that timeout is only relevant if the server is blocking on returning a response. If the request returns with error immediately then the timeout_seconds isn't really relevant.

[github-actions]

Hello! Thank you for this proposed change to cloud-init. This pull request is now marked as stale as it has not seen any activity in 14 days. If no activity occurs within the next 7 days, this pull request will automatically close.

If you are waiting for code review and you are seeing this message, apologies! Please reply, tagging TheRealFalcon, and he will ensure that someone takes a look soon.

(If the pull request is closed and you would like to continue working on it, please do tag TheRealFalcon to reopen it.)

[david-caro]

Oh yes, changing the config seemed to work, thanks!

I'll close the pr/issue