|
1 | | -.. warning:: |
| 1 | +This project is no longer maintained. |
2 | 2 |
|
3 | | - For very long, we recommended switching your CI jobs to `Zuulv3 |
4 | | - native jobs`__. In the Xena cycle, we are officially deprecating |
5 | | - it. Devstack Gate will support only stable branches until stable/wallaby. |
6 | | - From the Xena release onwards, we no longer guarantee it to work. If it |
7 | | - fails, we strongly recommend to switch your CI jobs to Zuulv3 native or |
8 | | - fork this repo to fix it for your CI. |
| 3 | +The contents of this repository are still available in the Git |
| 4 | +source code management system. To see the contents of this |
| 5 | +repository before it reached its end of life, please check out the |
| 6 | +previous commit with "git checkout HEAD^1". |
9 | 7 |
|
10 | | - We will retire it completely once stable/wallaby is in 'Extended Maintenance' |
11 | | - state which is 2022-10-14. |
| 8 | +Devstack is now capable of bootstrapping itself in CI environments |
| 9 | +using ansible playbooks and roles. You should not need devstack-gate |
| 10 | +to act as a driver. |
12 | 11 |
|
13 | | -.. __: https://governance.openstack.org/tc/goals/selected/victoria/native-zuulv3-jobs.html |
14 | | - |
15 | | -Devstack Gate |
16 | | -============= |
17 | | - |
18 | | -Devstack-gate is a collection of scripts used by the OpenStack CI team |
19 | | -to test every change to core OpenStack projects by deploying OpenStack |
20 | | -via devstack on a cloud server. |
21 | | - |
22 | | -What It Is |
23 | | -========== |
24 | | - |
25 | | -All changes to core OpenStack projects are "gated" on a set of tests |
26 | | -so that it will not be merged into the main repository unless it |
27 | | -passes all of the configured tests. Most projects require unit tests |
28 | | -with pep8 and several versions of Python. Those tests are all run only |
29 | | -on the project in question. The devstack gate test, however, is an |
30 | | -integration test and ensures that a proposed change still enables |
31 | | -several of the projects to work together. |
32 | | - |
33 | | -Obviously we test integrated OpenStack components and their clients |
34 | | -because they all work closely together to form an OpenStack |
35 | | -system. Changes to devstack itself are also required to pass this test |
36 | | -so that we can be assured that devstack is always able to produce a |
37 | | -system capable of testing the next change to nova. The devstack gate |
38 | | -scripts themselves are included for the same reason. |
39 | | - |
40 | | -How It Works |
41 | | -============ |
42 | | - |
43 | | -The devstack test starts with an essentially bare virtual machine, |
44 | | -installs devstack on it, and runs tests of the resulting OpenStack |
45 | | -installation. In order to ensure that each test run is independent, |
46 | | -the virtual machine is discarded at the end of the run, and a new |
47 | | -machine is used for the next run. In order to keep the actual test run |
48 | | -as short and reliable as possible, the virtual machines are prepared |
49 | | -ahead of time and kept in a pool ready for immediate use. The process |
50 | | -of preparing the machines ahead of time reduces network traffic and |
51 | | -external dependencies during the run. |
52 | | - |
53 | | -The `Nodepool`_ project is used to maintain this pool of machines. |
54 | | - |
55 | | -.. _Nodepool: https://opendev.org/zuul/nodepool |
56 | | - |
57 | | -How to Debug a Devstack Gate Failure |
58 | | -==================================== |
59 | | - |
60 | | -When Jenkins runs gate tests for a change, it leaves comments on the |
61 | | -change in Gerrit with a link to the resulting logs, including the |
62 | | -console log. If a change fails in a devstack-gate test, you can follow |
63 | | -these links to find out what went wrong. Start at the bottom of the log |
64 | | -file with the failure, scroll up to look for errors related to failed |
65 | | -tests. |
66 | | - |
67 | | -You might need some information about the specific run of the test. In |
68 | | -the devstack-gate-setup-workspace log, you can see all the git commands |
69 | | -used to set up the repositories, and they will output the (short) sha1 |
70 | | -and commit subjects of the head of each repository. |
71 | | - |
72 | | -It's possible that a failure could be a false negative related to a |
73 | | -specific provider, especially if there is a pattern of failures from |
74 | | -tests that run on nodes from that provider. In order to find out which |
75 | | -provider supplied the node the test ran on, look at the name of the |
76 | | -jenkins slave in the devstack-gate-setup-host log, the name of the |
77 | | -provider is included. |
78 | | - |
79 | | -Below that, you'll find the output from devstack as it installs all of |
80 | | -the debian and pip packages required for the test, and then configures |
81 | | -and runs the services. Most of what it needs should already be cached |
82 | | -on the test host, but if the change to be tested includes a dependency |
83 | | -change, or there has been such a change since the snapshot image was |
84 | | -created, the updated dependency will be downloaded from the Internet, |
85 | | -which could cause a false negative if that fails. |
86 | | -Assuming that there are no visible failures in the console log, you |
87 | | -may need to examine the log output from the OpenStack services, located |
88 | | -in the logs/ directory. All of the OpenStack services are configured to |
89 | | -syslog, so you may find helpful log messages by clicking on the |
90 | | -"syslog.txt[.gz]" file. Some error messages are so basic they don't |
91 | | -make it to syslog, such as if a service fails to start. Devstack |
92 | | -starts all of the services in screen, and you can see the output |
93 | | -captured by screen in files named "screen-\*.txt". You may find a |
94 | | -traceback there that isn't in syslog. |
95 | | - |
96 | | -After examining the output from the test, if you believe the result |
97 | | -was a false negative, you can retrigger the test by running a recheck, |
98 | | -this is done by leaving a review comment with simply the text: recheck |
99 | | - |
100 | | -If a test failure is a result of a race condition in the OpenStack code, |
101 | | -you also have the opportunity to try to identify it, and file a bug report, |
102 | | -help fix the problem or leverage `elastic-recheck |
103 | | -<http://docs.openstack.org/infra/elastic-recheck/readme.html>`_ to help |
104 | | -track the problem. If it seems to be related to a specific devstack gate |
105 | | -node provider, we'd love it if you could help identify what the variable |
106 | | -might be (whether in the devstack-gate scripts, devstack itself, Nodepool, |
107 | | -OpenStack, or even the provider's service). |
108 | | - |
109 | | -Simulating Devstack Gate Tests |
110 | | -============================== |
111 | | - |
112 | | -Developers often have a need to recreate gating integration tests |
113 | | -manually, and this provides a walkthrough of making a DG-slave-like |
114 | | -throwaway server without the overhead of building other CI |
115 | | -infrastructure to manage a pool of them. This can be useful to reproduce |
116 | | -and troubleshoot failures or tease out nondeterministic bugs. |
117 | | - |
118 | | -First, you can build an image identical to the images running in the gate using |
119 | | -`diskimage-builder <https://docs.openstack.org/developer/diskimage-builder>`_. |
120 | | -The specific operating systems built and DIB elements for each image type are |
121 | | -defined in `nodepool.yaml <https://opendev.org/openstack/project-config/ |
122 | | -src/branch/master/nodepool/nodepool.yaml>`_. There is a handy script |
123 | | -available in the project-config repo to build this for you:: |
124 | | - |
125 | | - git clone https://opendev.org/openstack/project-config |
126 | | - cd project-config |
127 | | - ./tools/build-image.sh |
128 | | - |
129 | | -Take a look at the documentation within the `build-image.sh` script for specific |
130 | | -build options. |
131 | | - |
132 | | -These days Tempest testing is requiring in excess of 2GiB RAM (4 should |
133 | | -be enough but we typically use 8) and completes within an hour on a |
134 | | -4-CPU virtual machine. |
135 | | - |
136 | | -If you're using an OpenStack provider, it's usually helpful to set up a |
137 | | -`clouds.yaml` file. More information on `clouds.yaml` files can be found in the |
138 | | -`os-client-config documentation <https://docs.openstack.org/developer/os-client-config/#config-files`_. |
139 | | -A `clouds.yaml` file for Rackspace would look something like:: |
140 | | - |
141 | | - |
142 | | - clouds: |
143 | | - rackspace: |
144 | | - auth: |
145 | | - profile: rackspace |
146 | | - username: '<provider_username>' |
147 | | - password: '<provider_password>' |
148 | | - project_name: '<provider_project_name>' |
149 | | - |
150 | | -Where provider_username and provider_password are the user / password |
151 | | -for a valid user in your account, and provider_project_name is the project_name |
152 | | -you want to use (sometimes called 'tenant name' on older clouds) |
153 | | - |
154 | | -You can then use the `openstack` command line client (found in the python |
155 | | -package |
156 | | -`python-openstackclient <http://pypi.python.org/pypi/python-openstackclient>`_) |
157 | | -to create a VM on the cloud. |
158 | | - |
159 | | -You can tell `openstack` to use the `DFW` region |
160 | | -of the `rackspace` cloud you defined either by setting environment variables:: |
161 | | - |
162 | | - export OS_CLOUD=rackspace |
163 | | - export OS_REGION_NAME=DFW |
164 | | - openstack servers list |
165 | | - |
166 | | -or command line options: |
167 | | - |
168 | | - openstack --os-cloud=rackspace --os-region-name=DFW servers list |
169 | | - |
170 | | -It will be assumed in remaining examples that environment varialbes have been |
171 | | -set. |
172 | | - |
173 | | -If you haven't already, create an SSH keypair "my-keypair" (name it whatever |
174 | | -you like):: |
175 | | - |
176 | | - openstack keypair create --public-key=$HOME/.ssh/id_rsa.pub my-keypair |
177 | | - |
178 | | -Upload your image, boot a server named "testserver" (chosen arbitrarily for |
179 | | -this example) with your SSH key allowed, and log into it:: |
180 | | - |
181 | | - FLAVOR='8GB Standard Instance' |
182 | | - openstack image create --file devstack-gate.qcow2 devstack-gate |
183 | | - openstack server create --wait --flavor "$FLAVOR" --image "devstack-gate" \ |
184 | | - --key-name=my-keypair testserver |
185 | | - openstack server ssh testserver |
186 | | - |
187 | | -If you get a cryptic error like ``ERROR: 'public'`` then you may need to |
188 | | -manually look up the IP address with ``openstack server show testserver`` and |
189 | | -connect by running ``ssh root@<ip_address>`` instead. Once logged in, switch to |
190 | | -the jenkins user and set up parts of the environment expected by devstack-gate |
191 | | -testing:: |
192 | | - |
193 | | - su - jenkins |
194 | | - export REPO_URL=https://git.openstack.org |
195 | | - export ZUUL_URL=/home/jenkins/workspace-cache |
196 | | - export ZUUL_REF=HEAD |
197 | | - export WORKSPACE=/home/jenkins/workspace/testing |
198 | | - mkdir -p $WORKSPACE |
199 | | - |
200 | | -Specify the project and branch you want to test for integration:: |
201 | | - |
202 | | - export ZUUL_PROJECT=openstack/nova |
203 | | - export ZUUL_BRANCH=master |
204 | | - |
205 | | -Get a copy of the tested project. After these steps, apply relevant |
206 | | -patches on the target branch (via cherry-pick, rebase, et cetera) and |
207 | | -make sure ``HEAD`` is at the ref you want tested:: |
208 | | - |
209 | | - git clone $REPO_URL/$ZUUL_PROJECT $ZUUL_URL/$ZUUL_PROJECT \ |
210 | | - && cd $ZUUL_URL/$ZUUL_PROJECT \ |
211 | | - && git checkout remotes/origin/$ZUUL_BRANCH |
212 | | - |
213 | | -Switch to the workspace and get a copy of devstack-gate:: |
214 | | - |
215 | | - cd $WORKSPACE \ |
216 | | - && git clone --depth 1 $REPO_URL/openstack/devstack-gate |
217 | | - |
218 | | -At this point you're ready to set the same environment variables and run |
219 | | -the same commands/scripts as used in the desired job. The definitions |
220 | | -for these are found in the openstack/project-config project under |
221 | | -the jenkins/jobs directory in a file named devstack-gate.yaml. It will |
222 | | -probably look something like:: |
223 | | - |
224 | | - export PYTHONUNBUFFERED=true |
225 | | - export DEVSTACK_GATE_TEMPEST=1 |
226 | | - export DEVSTACK_GATE_TEMPEST_FULL=1 |
227 | | - cp devstack-gate/devstack-vm-gate-wrap.sh ./safe-devstack-vm-gate-wrap.sh |
228 | | - ./safe-devstack-vm-gate-wrap.sh |
229 | | - |
230 | | -If you're trying to figure out which devstack gate jobs run for a given |
231 | | -project+branch combination, this is encoded in the |
232 | | -openstack/project-config project under the zuul/ directory in a file |
233 | | -named layout.yaml. You'll want to look in the "projects" section for a list |
234 | | -of jobs run on a given project in the "gate" pipeline, and then consult the |
235 | | -"jobs" section of the file to see if there are any overrides indicating |
236 | | -which branches qualify for the job and whether or not its voting is |
237 | | -disabled. |
238 | | - |
239 | | -After the script completes, investigate any failures. Then log out and |
240 | | -``openstack server delete testserver`` or similar to get rid of it once no |
241 | | -longer needed. It's possible to re-run certain jobs or specific tests on a used |
242 | | -VM (sometimes with a bit of manual clean-up in between runs), but for |
243 | | -proper testing you'll want to validate your fixes on a completely fresh |
244 | | -one. |
245 | | - |
246 | | -Refer to the `Jenkins Job Builder`_ and Zuul_ documentation for more |
247 | | -information on their configuration file formats. |
248 | | - |
249 | | -.. _`Jenkins Job Builder`: http://docs.openstack.org/infra/system-config/jjb.html |
250 | | - |
251 | | -.. _Zuul: http://docs.openstack.org/infra/system-config/zuul.html |
252 | | - |
253 | | -Contributions Welcome |
254 | | -===================== |
255 | | - |
256 | | -All of the OpenStack developer infrastructure is freely available and |
257 | | -managed in source code repositories just like the code of OpenStack |
258 | | -itself. If you'd like to contribute, just clone and propose a patch to |
259 | | -the relevant repository:: |
260 | | - |
261 | | - https://opendev.org/openstack/devstack-gate |
262 | | - https://opendev.org/zuul/nodepool |
263 | | - https://opendev.org/opendev/system-config |
264 | | - https://opendev.org/openstack/project-config |
265 | | - |
266 | | -You can file bugs on the storyboard devstack-gate project:: |
267 | | - |
268 | | - https://storyboard.openstack.org/#!/project/712 |
269 | | - |
270 | | -And you can chat with us on OFTC in #openstack-qa or #openstack-infra. |
271 | | - |
272 | | -It's worth noting that, while devstack-gate is generally licensed under the |
273 | | -Apache license, `playbooks/plugins/callback/devstack.py` is GPLv3 due to having |
274 | | -derived from the Ansible source code. |
| 12 | +For any further questions, please email |
| 13 | +openstack-discuss@lists.openstack.org or join #openstack-dev on |
| 14 | +OFTC. |
0 commit comments