qa/suites/fs: stop looping in mds upgrade test if upgrade failed

[adk3798]

Signed-off-by: Adam King adking@redhat.com

Testing for https://tracker.ceph.com/issues/54419

Also possibly something we want to actually merge depending on how the results go here. Having the test fail faster if the upgrade is failed for whatever reason would be a big improvement over running for 6 hours until the job is marked dead.

Checklist

• Tracker (select at least one)
  • References tracker ticket
  • Very recent bug; references commit where it was introduced
  • New feature (ticket optional)
  • Doc update (no ticket needed)
  • Code cleanup (no ticket needed)
• Component impact
  • Affects Dashboard, opened tracker ticket
  • Affects Orchestrator, opened tracker ticket
  • No impact that needs to be tracked
• Documentation (select at least one)
  • Updates relevant documentation
  • No doc update is appropriate
• Tests (select at least one)
  • Includes unit test(s)
  • Includes integration test(s)
  • Includes bug reproducer
  • No tests

Show available Jenkins commands

• jenkins retest this please
• jenkins test classic perf
• jenkins test crimson perf
• jenkins test signed
• jenkins test make check
• jenkins test make check arm64
• jenkins test submodules
• jenkins test dashboard
• jenkins test dashboard cephadm
• jenkins test api
• jenkins test docs
• jenkins render docs
• jenkins test ceph-volume all
• jenkins test ceph-volume tox
• jenkins test windows

[kotreshhr]

Makes sense. lgtm

[adk3798]

http://pulpito.front.sepia.ceph.com/adking-2022-03-11_23:05:59-orch:cephadm:mds_upgrade_sequence-wip-adk2-testing-2022-03-11-1538-distro-basic-smithi/

1 failure due to machine not being locked, unrelated infra issue
1 failure by https://tracker.ceph.com/issues/49287 (unrelated to this test in particular)

Didn't hit the actual upgrade failure I was looking for. Need another run.

[adk3798]

http://pulpito.front.sepia.ceph.com/adking-2022-03-14_12:43:51-orch:cephadm:mds_upgrade_sequence-wip-adk2-testing-2022-03-11-1538-distro-basic-smithi/

no failures

[batrick]

Suggest also just exit 1 after 30 minutes too. No reason to let this run for 12h if something gets stuck.

[adk3798]

added a 30 minute timeout on the command

EDIT: later removed it, as it was causing the test to be run with the old while statement for whatever reason. Will need to figure out a new way to do the timeout at some point.

[adk3798]

all passed

http://pulpito.front.sepia.ceph.com/adking-2022-03-15_17:49:30-orch:cephadm:mds_upgrade_sequence-wip-adk2-testing-2022-03-15-0949-distro-basic-smithi/

[adk3798]

jenkins test make check

[adk3798]

jenkins test make check

[adk3798]

@vshankar I haven't been able to actually hit the upgrade issue while testing this. What do you think about just merging it in then if you see the test fail again you can ping me so I can take a look? It will have more info I can use for debugging and will at least fail in 30 minutes rather than 6 hours.

[vshankar]

@vshankar I haven't been able to actually hit the upgrade issue while testing this. What do you think about just merging it in then if you see the test fail again you can ping me so I can take a look? It will have more info I can use for debugging and will at least fail in 30 minutes rather than 6 hours.

Absolutely. Let's merge this. I'll let you know how further tests look...

[vshankar]

jenkins test make check

[adk3798]

jenkins test make check

[adk3798]

Just noticing in this run http://pulpito.front.sepia.ceph.com/adking-2022-03-15_17:49:30-orch:cephadm:mds_upgrade_sequence-wip-adk2-testing-2022-03-15-0949-distro-basic-smithi/ after the timeout was added it seems to just be running the unchanged test

- cephadm.shell:
        env:
        - sha1
        host.a:
        - while ceph orch upgrade status | jq '.in_progress' | grep true ; do ceph orch

but pre timeout it took the changes http://pulpito.front.sepia.ceph.com/adking-2022-03-14_12:43:51-orch:cephadm:mds_upgrade_sequence-wip-adk2-testing-2022-03-11-1538-distro-basic-smithi/

    - cephadm.shell:
        env:
        - sha1
        host.a:
        - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch
          upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions
          ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done

It looks like with the timeout added to the front this doesn't actually work? It just ignored the changes entirely. Not sure how this works but at least it doesn't seem to like using timeout like this.

[vshankar]

@adk3798 Noticed that just now -- https://pulpito.ceph.com/vshankar-2022-03-16_09:42:54-fs:upgrade-wip-vshankar-testing-20220316-102808-testing-default-smithi/6739031/

I can't even see (in teuthology log) the grep Error string you introduced :/

[vshankar]

@adk3798 any progress on this?

[adk3798]

@adk3798 any progress on this?

@vshankar I removed the timeout bit for now. I think what's here will cover most of the cases where it's running for a long time anyhow and at the least it will give the info needed to figure out why it isn't finishing. I'd run this through tests again as it is now.

[adk3798]

http://pulpito.front.sepia.ceph.com/adking-2022-03-29_23:26:23-orch:cephadm-wip-adk-testing-2022-03-29-1402-distro-basic-smithi/

Failures:
https://tracker.ceph.com/issues/44587
https://tracker.ceph.com/issues/49287
1 machine failed to install kernel fast enough

[adk3798]

jenkins test make check

[adk3798]

@vshankar do we want to merge this?

[vshankar]

@vshankar do we want to merge this?

I'm ok with this change if the run fine.