Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make anexia provider more resilient against errors #1175

Conversation

kstiehl
Copy link
Contributor

@kstiehl kstiehl commented Jan 25, 2022

Signed-off-by: kstiehl kevin.stiehl@numericas.de

What this PR does / why we need it:

The Anexia API can behave a little bit weird sometimes.

  • Requests to the API get a negative status code because of some internal engine timeouts but in the VM get's provisionend anyways. We are now tracking whether we already provisionend an IP, and stop provisioning more of them on error.
  • During deletion we use the new deprovisioning tracking
  • The machine controller remembers ongoing provisionings and reattaches to them now after an error

Which issue(s) this PR fixes (optional, in fixes #<issue number> format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

Optional Release Note:

Anexia Provider is more resilient now and supports reattaching to ongoing provisionings after an error

@kubermatic-bot kubermatic-bot added dco-signoff: yes Denotes that all commits in the pull request have the valid DCO signoff message. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. sig/cluster-management Denotes a PR or issue as being assigned to SIG Cluster Management. sig/osm Denotes a PR or issue as being assigned to SIG OSM. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jan 25, 2022
@kubermatic-bot
Copy link
Contributor

Hi @kstiehl. Thanks for your PR.

I'm waiting for a kubermatic member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@kubermatic-bot kubermatic-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Jan 25, 2022
@moadqassem
Copy link
Member

@kstiehl Thank you very much for your effort. Can you please rebase your changes so we can start the reviewing process? Cheers!

@LittleFox94
Copy link
Contributor

@moadqassem just randomly saw your request and will do that on @kstiehl's behalf, as he does not work for us anymore - commits will say unverified here on github though

@github-actions github-actions bot force-pushed the for-upstream/fix/make-machine-controller-more-resilient branch from 6b52096 to 5a4d0de Compare March 16, 2022 15:37
@ahmedwaleedmalik
Copy link
Member

/retest

@ahmedwaleedmalik
Copy link
Member

@kstiehl can you please refactor the PR to take care of:

  • pull-machine-controller-lint
  • pull-machine-controller-test
  • pull-machine-controller-verify-boilerplate

Other failures are not related to your PR and I'm taking a look at them.

@LittleFox94
Copy link
Contributor

@ahmedwaleedmalik I'll look into it

@github-actions github-actions bot force-pushed the for-upstream/fix/make-machine-controller-more-resilient branch 2 times, most recently from 63c94c2 to eb7ff85 Compare March 22, 2022 10:53
@kubermatic-bot kubermatic-bot added dco-signoff: no Denotes that at least one commit in the pull request doesn't have a valid DCO signoff message. and removed dco-signoff: yes Denotes that all commits in the pull request have the valid DCO signoff message. labels Mar 22, 2022
@github-actions github-actions bot force-pushed the for-upstream/fix/make-machine-controller-more-resilient branch from eb7ff85 to 7ba9678 Compare March 22, 2022 10:55
@kubermatic-bot kubermatic-bot added dco-signoff: yes Denotes that all commits in the pull request have the valid DCO signoff message. and removed dco-signoff: no Denotes that at least one commit in the pull request doesn't have a valid DCO signoff message. labels Mar 22, 2022
@github-actions github-actions bot force-pushed the for-upstream/fix/make-machine-controller-more-resilient branch from 7ba9678 to 3bdcbd8 Compare March 22, 2022 11:01
kstiehl and others added 4 commits March 22, 2022 11:32
Original commit by @kstiehl, conflicts resolved by @LittleFox94, mainly
in pkg/cloudprovider/provider/anexia/provider.go

Signed-off-by: kstiehl <kevin.stiehl@numericas.de>
Signed-off-by: Mara Sophie Grosch <mgrosch@anexia-it.com>
Signed-off-by: Mara Sophie Grosch <mgrosch@anexia-it.com>
Signed-off-by: Mara Sophie Grosch <mgrosch@anexia-it.com>
Signed-off-by: Mara Sophie Grosch <mgrosch@anexia-it.com>
@github-actions github-actions bot force-pushed the for-upstream/fix/make-machine-controller-more-resilient branch from 3bdcbd8 to c849820 Compare March 22, 2022 11:32
@ahmedwaleedmalik
Copy link
Member

/ok-to-test

@kubermatic-bot kubermatic-bot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Mar 22, 2022
@kubermatic-bot kubermatic-bot removed the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Mar 22, 2022
Copy link
Member

@ahmedwaleedmalik ahmedwaleedmalik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve

@kubermatic-bot kubermatic-bot added the lgtm Indicates that a PR is ready to be merged. label Mar 22, 2022
@kubermatic-bot
Copy link
Contributor

LGTM label has been added.

Git tree hash: 1ff205f7475055b3b4c75308052364384da1a9db

@kubermatic-bot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahmedwaleedmalik, kstiehl

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubermatic-bot kubermatic-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 22, 2022
@kubermatic-bot kubermatic-bot merged commit b067fb6 into kubermatic:master Mar 22, 2022
@LittleFox94 LittleFox94 deleted the for-upstream/fix/make-machine-controller-more-resilient branch March 23, 2022 11:32
LittleFox94 added a commit to anexia-it/machine-controller that referenced this pull request Aug 15, 2022
* add conditions & tests

Original commit by @kstiehl, conflicts resolved by @LittleFox94, mainly
in pkg/cloudprovider/provider/anexia/provider.go

Signed-off-by: kstiehl <kevin.stiehl@numericas.de>
Signed-off-by: Mara Sophie Grosch <mgrosch@anexia-it.com>

* Fix various linter errors

Signed-off-by: Mara Sophie Grosch <mgrosch@anexia-it.com>

* Add missing boilerplate to files

Signed-off-by: Mara Sophie Grosch <mgrosch@anexia-it.com>

* Anexia provider: add dummy updater for tests

Signed-off-by: Mara Sophie Grosch <mgrosch@anexia-it.com>

Co-authored-by: Mara Sophie Grosch <mgrosch@anexia-it.com>
kubermatic-bot pushed a commit that referenced this pull request Aug 23, 2022
* Make anexia provider more resilient against errors (#1175)

* add conditions & tests

Original commit by @kstiehl, conflicts resolved by @LittleFox94, mainly
in pkg/cloudprovider/provider/anexia/provider.go

Signed-off-by: kstiehl <kevin.stiehl@numericas.de>
Signed-off-by: Mara Sophie Grosch <mgrosch@anexia-it.com>

* Fix various linter errors

Signed-off-by: Mara Sophie Grosch <mgrosch@anexia-it.com>

* Add missing boilerplate to files

Signed-off-by: Mara Sophie Grosch <mgrosch@anexia-it.com>

* Anexia provider: add dummy updater for tests

Signed-off-by: Mara Sophie Grosch <mgrosch@anexia-it.com>

Co-authored-by: Mara Sophie Grosch <mgrosch@anexia-it.com>

* docs(anexia): add info for templates (#1288)

* docs(anexia): add info for templates

Signed-off-by: Waleed Malik <ahmedwaleedmalik@gmail.com>

* Update anexia.md

Signed-off-by: Waleed Malik <ahmedwaleedmalik@gmail.com>

* anexia: increase HTTP client timeout to 120s (#1331)

Signed-off-by: Mara Sophie Grosch <mgrosch@anexia-it.com>

* Anexia: extend disk configuration (#1402)

* Anexia: cleanup Config/RawConfig attribute duplication

types.Config and types.RawConfig contained the same members, with
RawConfig using ConfigVars and Config storing the resolved data. This
commit does some cleanup to ease adding new config values in the future.

Signed-off-by: Mara Sophie Grosch <mgrosch@anexia-it.com>

* Anexia: extend disk configuration

Adds a new config value "Disks", with each entry configuring the size
and performance type of a single disk. The config format can be used for
configuring multiple disks in the future, which is on our roadmap, but
right now only a single disk is supported - with the added feature of
defining the performance type.

Signed-off-by: Mara Sophie Grosch <mgrosch@anexia-it.com>

* Anexia: update MachineDeployment example

Signed-off-by: Mara Sophie Grosch <mgrosch@anexia-it.com>

* CI: disable OperationSystemManager for Anexia

Signed-off-by: Mara Sophie Grosch <mgrosch@anexia-it.com>

Signed-off-by: Mara Sophie Grosch <mgrosch@anexia-it.com>

* Anexia: return reserved IP when instance IPs are requested

When creating a machine at, we always first reserve an IP address for
it. Later when retrieving the addresses of that machine, we rely on the
vminfo API, which has some delay as it retrieves the IPs from the
running VM.

This commit adds the reserved address to the list of addresses returned
from an instance, which should reduce provisioning time a bit and make
it more stable.

Also fixes a long-standing TODO comment: marking internal IPs as
internal.

Signed-off-by: Mara Sophie Grosch <mgrosch@anexia-it.com>

Signed-off-by: Waleed Malik <ahmedwaleedmalik@gmail.com>
Signed-off-by: Mara Sophie Grosch <mgrosch@anexia-it.com>
Co-authored-by: Kevin Stiehl <kevinstiehl@live.de>
Co-authored-by: Waleed Malik <ahmedwaleedmalik@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. dco-signoff: yes Denotes that all commits in the pull request have the valid DCO signoff message. lgtm Indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/cluster-management Denotes a PR or issue as being assigned to SIG Cluster Management. sig/osm Denotes a PR or issue as being assigned to SIG OSM. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants