Nova fails when running Atmosphere in dev env.

[mpiscaer]

When running the converge Nova fails.

root@ctl1:/var/log# kubectl -n openstack get pods|grep nova
nova-api-metadata-fcc5c7b9c-cmsdd 0/1 Init:0/2 0 13m
nova-api-metadata-fcc5c7b9c-kqvmf 0/1 Init:0/2 0 13m
nova-api-metadata-fcc5c7b9c-pghct 0/1 Init:0/2 0 13m
nova-api-osapi-7bdddd9c49-2ggn2 0/1 Init:0/1 0 13m
nova-api-osapi-7bdddd9c49-qgg9r 0/1 Init:0/1 0 13m
nova-api-osapi-7bdddd9c49-tt4w8 0/1 Init:0/1 0 13m
nova-bootstrap-qs5bs 0/1 Init:0/1 0 13m
nova-cell-setup-zbhs2 0/1 Init:0/2 0 13m
nova-compute-default-d6ftj 0/2 Init:0/7 0 13m
nova-compute-default-zgzxx 0/2 Init:0/7 0 13m
nova-conductor-74b7d47f7-b4ff5 0/1 Init:0/1 0 13m
nova-conductor-74b7d47f7-rvxrr 0/1 Init:0/1 0 13m
nova-conductor-74b7d47f7-svnv6 0/1 Init:0/1 0 13m
nova-db-init-szkvw 0/3 Completed 0 13m
nova-db-sync-s5qh8 0/1 Completed 0 13m
nova-novncproxy-66f9444464-5l8dp 1/1 Running 0 13m
nova-novncproxy-66f9444464-hnd99 1/1 Running 0 13m
nova-novncproxy-66f9444464-pf52z 1/1 Running 0 13m
nova-scheduler-66f86545cc-6rbs4 0/1 Init:0/1 0 13m
nova-scheduler-66f86545cc-jj8qk 0/1 Init:0/1 0 13m
nova-scheduler-66f86545cc-sjvsc 0/1 Init:0/1 0 13m
rabbitmq-nova-server-0 1/1 Running 0 29m

nova-db-sync-s5qh8 takes longer then 6 minutes to complete.

root@ctl1:# kubectl -n openstack get pods|grep sync
< snip >
nova-db-sync-s5qh8 1/1 Running 0 5m44s
root@ctl1:# kubectl -n openstack get pods|grep sync
< snip >
nova-db-sync-s5qh8 0/1 Completed 0 7m27s

After it completes, I view the logs of nova-api-osapi-7bdddd9c49-tt4w8:

Entrypoint WARNING: 2022/10/28 08:00:07 entrypoint.go:72: Resolving dependency Job nova-rabbit-init in namespace openstack failed: jobs.batch "nova-rabbit-init" not found .
Entrypoint WARNING: 2022/10/28 08:00:07 entrypoint.go:72: Resolving dependency Job nova-ks-endpoints in namespace openstack failed: jobs.batch "nova-ks-endpoints" not found .
Entrypoint WARNING: 2022/10/28 08:00:07 entrypoint.go:72: Resolving dependency Job nova-ks-user in namespace openstack failed: jobs.batch "nova-ks-user" not found .

So nova-rabbit-init is missing and it never comes up. The only workaround what I found is doing a helm -n openstack uninstall nova and then rerun the playbook.

But this is not a option in my CI process.

Kind regards,

Michiel Piscaer

[mnaser]

It looks like it's taking way more than 10 minutes to run your Nova deployment... that seems odd.

Can you check what is the output of the following?

kubectl -n openstack get helmrelease

[mpiscaer]

I did a ceph benchmark, but I don't know what to think about it. This runs ceph on ceph on nvme.

Average IOPS: 35 does not sounds alot.

ceph osd pool create scbench 128 128

root@ctl1:# rados bench -p scbench 10 write --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_ctl1_330449
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 16 16 0 0 0 - 0
1 16 44 28 111.923 112 0.373966 0.466264
2 16 79 63 125.947 140 0.699742 0.450056
3 16 120 104 138.622 164 0.623496 0.430145
4 16 159 143 142.96 156 0.250668 0.412846
5 16 197 181 144.763 152 0.349505 0.421202
6 16 223 207 137.967 104 0.207419 0.414248
7 16 255 239 136.54 128 0.653638 0.452057
8 16 299 283 141.469 176 0.511271 0.445253
9 16 330 314 139.527 124 0.729013 0.442047
10 16 371 355 141.972 164 0.4878 0.440695
Total time run: 10.4376
Total writes made: 371
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 142.179
Stddev Bandwidth: 24.2395
Max bandwidth (MB/sec): 176
Min bandwidth (MB/sec): 104
Average IOPS: 35
Stddev IOPS: 6.05989
Max IOPS: 44
Min IOPS: 26
Average Latency(s): 0.438977
Stddev Latency(s): 0.224824
Max latency(s): 1.47564
Min latency(s): 0.0946544

root@ctl1:# rados bench -p scbench 10 seq
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 16 16 0 0 0 - 0
1 16 64 48 191.163 192 0.0601915 0.244665
2 16 126 110 219.488 248 0.186607 0.237952
3 16 202 186 247.603 304 0.063237 0.240049
4 16 269 253 252.687 268 0.0807225 0.231722
5 16 342 326 260.536 292 0.0500561 0.235051
Total time run: 5.73865
Total reads made: 371
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 258.597
Average IOPS: 64
Stddev IOPS: 11.0318
Max IOPS: 76
Min IOPS: 48
Average Latency(s): 0.237449
Max latency(s): 1.12343
Min latency(s): 0.0292632
root@ctl1:~# rados bench -p scbench 10 rand
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 16 16 0 0 0 - 0
1 16 83 67 267.874 268 0.554064 0.194011
2 16 154 138 275.919 284 0.662703 0.188191
3 16 222 206 273.32 272 0.128474 0.190124
4 16 301 285 283.941 316 0.226596 0.184738
5 15 370 355 283.032 280 0.271027 0.21768
6 16 450 434 288.507 316 0.475762 0.214854
7 16 531 515 293.555 324 0.0392132 0.210433
8 15 608 593 295.843 312 0.215073 0.211374
9 16 694 678 300.717 340 0.0896654 0.207491
10 15 764 749 299.045 284 0.710712 0.207093
Total time run: 10.3155
Total reads made: 764
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 296.253
Average IOPS: 74
Stddev IOPS: 6.20842
Max IOPS: 85
Min IOPS: 67
Average Latency(s): 0.212006
Max latency(s): 3.44772
Min latency(s): 0.0160714

root@ctl1:# kubectl -n openstack get helmrelease
NAME AGE READY STATUS
barbican 58m True Release reconciliation succeeded
ceph-provisioners 57m True Release reconciliation succeeded
cinder 45m True Release reconciliation succeeded
glance 57m True Release reconciliation succeeded
ingress-nginx 58m True Release reconciliation succeeded
keystone 58m True Release reconciliation succeeded
libvirt 45m True Release reconciliation succeeded
memcached 58m True Release reconciliation succeeded
neutron 45m True Release reconciliation succeeded
neutron-coredns 45m True Release reconciliation succeeded
nova 11m False install retries exhausted
openvswitch 45m True Release reconciliation succeeded
placement 45m True Release reconciliation succeeded
pxc-operator 58m True Release reconciliation succeeded
rabbitmq-cluster-operator 56m True Release reconciliation succeeded

[mpiscaer]

helmrelease-nova.txt

kubectl -n openstack describe helmrelease nova

[mnaser]

This is not really that relevant and could be simply related to #210