openstack backend by sergiocazzolato · Pull Request #170 · canonical/spread

sergiocazzolato · 2023-06-02T16:57:52Z

This is the new openstack backend. Through this backend it is possible to run tests/tasks in openstack infrastructure.

The documenatation is also added explaning how to setup spread to use it.

For the openstack backend implementation the lib goose was used. This lib provides the clients needed to interact with the different openstack modules (nova. neutron, glance and, keystone).

corytodd · 2023-06-07T19:15:01Z

+	}
+	server, err := p.computeClient.RunServer(opts)
+	if err != nil {
+		return nil, &FatalError{fmt.Errorf("Could not create instance", err)}


Should the format string include a format specifier here?

the FatalError means it is not needed to retry and it is built from an error, I already updated the error in the backend to make sure we retry only when it makes sense

Oh okay, I see know thanks for explaining.

corytodd · 2023-06-07T19:22:00Z

Awesome, we are able to allocate and discard on our openstack tenant. Do you think it's feasible to support adding security groups automatically? For example, we do not use a default allow-ssh policy on our network so in order for these tests to work I had to manually attach the policy after allocation but before the test ran. We have other ports that may need to be open depending on the test so having a way to set these dynamically would be useful for us.

sergiocazzolato · 2023-06-07T19:33:18Z

Awesome, we are able to allocate and discard on our openstack tenant. Do you think it's feasible to support adding security groups automatically? For example, we do not use a default allow-ssh policy on our network so in order for these tests to work I had to manually attach the policy after allocation but before the test ran. We have other ports that may need to be open depending on the test so having a way to set these dynamically would be useful for us.

Thanks for taking a look. I'll add few extra features:
add extra storage
specify the network
specify the security group

corytodd · 2023-06-09T20:45:47Z

Specifying the network and security group work great! This passes our testing, I would support a +1 on getting this merged. Thanks!

mvo5

Thanks for working on this Sergio! I did a first review and have some suggestion inline. I will also sync with Gustavo to ask how he wants to see this moving forward.

mvo5 · 2023-07-04T15:27:48Z

+		return sameImage, fmt.Errorf("failed to retrieve images list: %s", errorTitle(err.Error()))
+	}
+
+	for _, i := range images {


I wonder if this code should be slightly more elaborate and follow googleProvider:image or linode:tempate(). linode is simpler and just does a prefix search but afaict all do more than just check for "contains"

@mvo5 could you please elaborate a bit more this? In openstack the images dont have family or project associated as in gce, so because of that I used the contain.

Sorry, I was mostly wondering what contraints there are about image names, I created a unit test for the code now so that we can explore various test cases and examples :)

mvo5

Gustavo also asked that backends that do not (yet) support the options network/groups should error when they are specified.

A smoke spread test against a real system should be included and unit tests as far as possible without modifying non-openstack code.

The way images are selected/filtered also needs a review.

sergiocazzolato · 2023-07-17T17:43:22Z

@mvo5 about the network list associated to a machine, the consideration here is that where there are more than 1 network associated to a machine, the ip used by spread to connect has to be provided by the first network. I'll include that in the README.

…tack

mvo5

Thanks, I looekd a bit more and I really like the updated spread test! I also added a few comments and suggestions and pushed some small tweaks.

mvo5 · 2023-08-17T11:31:50Z

+		return sameImage, fmt.Errorf("failed to retrieve images list: %s", errorTitle(err.Error()))
+	}
+
+	for _, i := range images {


Sorry, I was mostly wondering what contraints there are about image names, I created a unit test for the code now so that we can explore various test cases and examples :)

…again

Also include test to validate the key/secret authentication and tne env file

ZeyadYasser

Thank you, small comments

Signed-off-by: Zeyad Gouda <zeyad.gouda@canonical.com>

devstack has many issues and cannot fully replicate a normal openstack cluster for testing. Signed-off-by: Zeyad Gouda <zeyad.gouda@canonical.com>

ZeyadYasser · 2024-03-25T21:55:57Z

I dropped the spread test for openstack due to issues and inconsistencies faced with devstack where it cannot replicate a normal openstack cluster.

A better alternative is to do something similar to google. After the openstack backend is merged, we add it as a backend in spread.yaml.

Now the volumes are cleaned manually to avoid issues when using delete_on_termination=true Garbage collection also is update to delete orphan volumes

Establish a set of attributes to encode credentials instead of reading them directly from the environment. Only identity API version 3 is currently supported. Signed-off-by: Claudio Matsuoka <claudio.matsuoka@canonical.com>

dariuszd21

Looks good to me, just minor questions

Signed-off-by: Claudio Matsuoka <claudio.matsuoka@canonical.com>

This reverts commit ff94632.

Signed-off-by: Claudio Matsuoka <claudio.matsuoka@canonical.com>

niemeyer

Thank you very much to everyone involved in developing and polishing this backend, and thanks @cmatsuoka for polishing the final details for unblocking it. With your latest changes, the shared part of these changes seem to follow the existing patterns, and any remaining problems are going to be specific to OpenStack itself, so let's go ahead and get this in.

There's one final detail to be done shortly in a follow up, which is fixing the casing, per comment below. But this is fine to be done as a separate PR so we don't wait even longer than we already did.

niemeyer · 2025-04-14T17:40:13Z

 [LXD backend](#lxd)  
 [QEMU backend](#qemu)  
 [Google backend](#google)  
+[Openstack backend](#openstack)  


The casing of the project name is incorrect, both in documentation and in the code itself. The project seems to call itself as OpenStack instead of Openstack, so I would suggest we follow suit and do a s/Openstack/OpenStack/ everywhere. Let's please keep "openstack", though (not "openStack").

sergiocazzolato added 3 commits April 14, 2023 12:38

New openstack backend

b4dfab2

First change for the openstack backend implementation

99a6884

Changes done to support openstack bakend after testing in canonistack

ccd286b

corytodd reviewed Jun 7, 2023

View reviewed changes

adding network and security groups to systems

81ec212

sergiocazzolato changed the title ~~tests: new openstack backend~~ openstack backend Jun 14, 2023

mvo5 reviewed Jul 4, 2023

View reviewed changes

mvo5 added 2 commits July 5, 2023 14:38

openstack: tweak function returns and messages

6adfc24

spread: rever change in {google,humbox,lindode}.go

b08c9ca

mvo5 force-pushed the migrate-openstack-backend branch from 04aa498 to b08c9ca Compare July 5, 2023 12:41

openstack: two more tweak about error message

e0052c3

This comment was marked as resolved.

Sign in to view

mvo5 reviewed Jul 5, 2023

View reviewed changes

Comment thread spread/project.go Outdated

mvo5 reviewed Jul 5, 2023

View reviewed changes

Comment thread spread/project.go Outdated

This comment was marked as outdated.

Sign in to view

mvo5 reviewed Jul 12, 2023

View reviewed changes

Comment thread spread/project.go Outdated

Comment thread spread/project.go Outdated

sergiocazzolato added 2 commits July 19, 2023 16:18

support networks and rename groups intead of security-groups in opens…

c82e63f

…tack

Add new test to validate openstack backend

b741ae6

sergiocazzolato requested a review from mvo5 July 22, 2023 03:00

sergiocazzolato and others added 4 commits July 29, 2023 16:25

Add new checks for errors and improve error management and messages

ba946a3

Test spread -gc

5e22375

openstack: add unit test for "openstackName()"

5c7f2bb

openstack: add unit tests for findImage()

0e4f033

mvo5 reviewed Aug 17, 2023

View reviewed changes

spread: improve openstackName() test and make it similar to "google" …

8fbae16

…again

mvo5 force-pushed the migrate-openstack-backend branch from 9053022 to 8fbae16 Compare August 17, 2023 16:28

sergiocazzolato added 2 commits March 12, 2024 22:31

Allow authentication with key and secret

5f673c8

Also include test to validate the key/secret authentication and tne env file

add more details in the openstack test to explain -gc

755dc0e

ZeyadYasser removed the Ready label Mar 20, 2024

ZeyadYasser reviewed Mar 20, 2024

View reviewed changes

Comment thread README.md Outdated

Comment thread spread.yaml Outdated

Comment thread spread.yaml

Comment thread spread/openstack.go Outdated

Comment thread spread/openstack.go

Comment thread spread/openstack.go

ZeyadYasser added 4 commits March 25, 2024 23:37

Fix README formatting

86b398d

Signed-off-by: Zeyad Gouda <zeyad.gouda@canonical.com>

openstack: comment out existing ssh configs that conflicts

701600c

Signed-off-by: Zeyad Gouda <zeyad.gouda@canonical.com>

openstack: rename goose{C,c}lient package

027ba48

Signed-off-by: Zeyad Gouda <zeyad.gouda@canonical.com>

openstack: drop spread tests

58c7fad

devstack has many issues and cannot fully replicate a normal openstack cluster for testing. Signed-off-by: Zeyad Gouda <zeyad.gouda@canonical.com>

Fix cloud-init config to make openstack work again

7f84982

niemeyer removed the squash-merge label Dec 9, 2024

sergiocazzolato and others added 3 commits January 15, 2025 11:30

Update the volumes management

ff94632

Now the volumes are cleaned manually to avoid issues when using delete_on_termination=true Garbage collection also is update to delete orphan volumes

Merge branch 'master' into migrate-openstack-backend

5277fb3

refactor(openstack): read credentials from backend attributes

2e787a8

Establish a set of attributes to encode credentials instead of reading them directly from the environment. Only identity API version 3 is currently supported. Signed-off-by: Claudio Matsuoka <claudio.matsuoka@canonical.com>

dariuszd21 approved these changes Feb 27, 2025

View reviewed changes

Comment thread spread/openstack.go Outdated

Comment thread spread/openstack.go

cmatsuoka force-pushed the migrate-openstack-backend branch 2 times, most recently from 5e21ff2 to 2e787a8 Compare March 11, 2025 13:12

refactor: improve authentication error handling

5184438

Signed-off-by: Claudio Matsuoka <claudio.matsuoka@canonical.com>

cmatsuoka reviewed Mar 11, 2025

View reviewed changes

Comment thread spread/openstack.go Outdated

cmatsuoka added 8 commits March 12, 2025 11:06

Revert "Update the volumes management"

e42b204

This reverts commit ff94632.

chore: misc post-revert adjustments

0dbfe70

Signed-off-by: Claudio Matsuoka <claudio.matsuoka@canonical.com>

tests: adjust openstack test patterns

731c1fc

Signed-off-by: Claudio Matsuoka <claudio.matsuoka@canonical.com>

chore: test adjustments

c953225

Signed-off-by: Claudio Matsuoka <claudio.matsuoka@canonical.com>

docs: update openstack backend in readme file

473ddd1

Signed-off-by: Claudio Matsuoka <claudio.matsuoka@canonical.com>

feat: obtain the identity api version from endpoint

bc4c81a

Signed-off-by: Claudio Matsuoka <claudio.matsuoka@canonical.com>

fix: don't probe unsupported api versions

a158f75

Signed-off-by: Claudio Matsuoka <claudio.matsuoka@canonical.com>

fix: close the identity response

c780387

Signed-off-by: Claudio Matsuoka <claudio.matsuoka@canonical.com>

cmatsuoka requested a review from niemeyer March 27, 2025 22:17

niemeyer approved these changes Apr 14, 2025

View reviewed changes

niemeyer merged commit f1035e9 into canonical:master Apr 14, 2025
1 check passed

Conversation

sergiocazzolato commented Jun 2, 2023

Uh oh!

corytodd Jun 7, 2023

Choose a reason for hiding this comment

Uh oh!

sergiocazzolato Jun 9, 2023

Choose a reason for hiding this comment

Uh oh!

corytodd Jun 9, 2023

Choose a reason for hiding this comment

Uh oh!

corytodd commented Jun 7, 2023

Uh oh!

sergiocazzolato commented Jun 7, 2023

Uh oh!

corytodd commented Jun 9, 2023

Uh oh!

mvo5 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mvo5 Jul 4, 2023

Choose a reason for hiding this comment

Uh oh!

sergiocazzolato Jul 18, 2023

Choose a reason for hiding this comment

Uh oh!

mvo5 Aug 17, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

mvo5 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sergiocazzolato commented Jul 17, 2023

Uh oh!

mvo5 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mvo5 Aug 17, 2023

Choose a reason for hiding this comment

Uh oh!

ZeyadYasser left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ZeyadYasser commented Mar 25, 2024

Uh oh!

dariuszd21 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

niemeyer left a comment

niemeyer Apr 14, 2025 •

edited

Loading