mgr/cephadm: Adding image tag and date to cephadm startup messages #45598

rkachach · 2022-03-23T17:31:16Z

This PR has two goals:

Ensure the image used by cephadm shell is in sync with any ceph daemon (mon, mgr, osd, etc) running on the local host
Show more information to the user to make him aware which image cephadm shell has chosen

The first point will help to fix the version mismatch issues reported in several trackers (specially after upgrade scenarios). Meanwhile the second point is an improvement of the current info shown to the user.

Previously when cephadm is launched we print something like:

[root@ceph-node-0 ~]# cephadm shell
Inferring fsid d71c2bbe-aabb-11ec-8403-52540052aca1
Using recent ceph image registry.hub.docker.com/rkachach/ceph@sha256:565277a8bcb13d2e123d444c4e5433cd28b221ec48fb97d99e4fcaae2233cf38

With the changes now it shows:

[root@ceph-node-0 ~]# cephadm shell
Inferring fsid f8e22d2c-bfd1-11ec-ad4a-525400120be8
Using ceph image with id '666bbfa87e8d' and tag 'custom-v0.5' created on 2022-02-02 12:03:03 +0000 UTC
registry.hub.docker.com/rkachach/ceph@sha256:1cb40d8f783f9725b67a1db97e615f4ff4355467fe3b793d489d33d562460ee8

Basically we add the fields id '666bbfa87e8d', tag 'v15.2.5' and created on 2020-09-16 14:15:15 +0000 UTC
show both tag and creation date this way it's easier to know which image cephadm is using.

Fixes: https://tracker.ceph.com/issues/55008
Fixes: https://tracker.ceph.com/issues/54373
Fixes: https://tracker.ceph.com/issues/54475
Signed-off-by: Redouane Kachach rkachach@redhat.com

Checklist

Tracker (select at least one)
- References tracker ticket
- Very recent bug; references commit where it was introduced
- New feature (ticket optional)
- Doc update (no ticket needed)
- Code cleanup (no ticket needed)
Component impact
- Affects Dashboard, opened tracker ticket
- Affects Orchestrator, opened tracker ticket
- No impact that needs to be tracked
Documentation (select at least one)
- Updates relevant documentation
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)
- Includes integration test(s)
- Includes bug reproducer
- [] No tests

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard cephadm
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox
jenkins test windows

rkachach · 2022-03-29T14:41:25Z

jenkins test make check

adk3798 · 2022-04-04T13:06:27Z

http://pulpito.ceph.com/adking-2022-04-01_16:41:38-orch:cephadm-wip-adk-testing-2022-04-01-0813-distro-basic-smithi/

1 failure caused by https://tracker.ceph.com/issues/49287

rkachach · 2022-04-05T07:50:51Z

http://pulpito.ceph.com/adking-2022-04-01_16:41:38-orch:cephadm-wip-adk-testing-2022-04-01-0813-distro-basic-smithi/

1 failure caused by https://tracker.ceph.com/issues/49287

This issue is caused by an old version of runc (1.0.0). I have already commented this with David but he has not updated the OS version of the tests node yet.

adk3798 · 2022-04-11T13:55:47Z

http://pulpito.front.sepia.ceph.com/adking-2022-04-08_22:17:12-orch:cephadm-wip-adk-testing-2022-04-08-1434-distro-basic-smithi/

2 failures:

1 caused by https://tracker.ceph.com/issues/49287
workunits/cephadm/test_cephadm.sh failure caused by another PR in this run.

adk3798 · 2022-04-13T15:50:02Z

http://pulpito.ceph.com/adking-2022-04-13_05:16:48-orch:cephadm-wip-adk-testing-2022-04-12-2231-distro-basic-smithi/

2 Failures

https://tracker.ceph.com/issues/44587
ceph-volume lvm batch failure ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-3/: (5) Input/output error that may be a faulty hardware thing, we're not sure. Did a rerun of the osds tests https://pulpito.ceph.com/adking-2022-04-13_13:12:12-orch:cephadm:osds-wip-adk-testing-2022-04-12-2231-distro-basic-smithi/ and the test that failed in the last run passed and the one test that did fail is due to https://tracker.ceph.com/issues/49287

To summarize, run was passing. Will start tracking this type of lvm batch failure to see if it happens on the same hosts (which would imply a possible hardware issue) in https://tracker.ceph.com/issues/55319

mgfritch · 2022-04-13T17:13:56Z

src/cephadm/cephadm

+        if image and not image.isspace():
+            (digest, tag, created_date) = image.lstrip().split('|')
+            if digest and not digest.endswith('@'):
+                logger.info(f"Using ceph image with tag '{tag}' created on {created_date}\n{digest}")


maybe the container image id would be more useful than then the creation time?

Updated the PR to add the image_id plus other important enhancements.

adk3798 · 2022-04-15T14:42:26Z

https://pulpito.ceph.com/adking-2022-04-15_05:56:09-orch:cephadm-wip-adk-testing-2022-04-14-1844-distro-basic-smithi/

1 infra failure reimaging machines

phlogistonjohn

Code looks OK to me, although I do find the unit test a tad confusing. I have no strong opinion about including the creation time.

src/cephadm/cephadm

src/cephadm/tests/test_cephadm.py

src/cephadm/cephadm

adk3798

I think this functionally works. Mostly just comments on tests, code comments, etc.

src/cephadm/cephadm

src/cephadm/tests/test_cephadm.py

src/cephadm/cephadm

phlogistonjohn

A few minor things mostly but I am concerned with the change to the eq behavior. Also, minor but the fixes in the 2nd patch should probably be squashed into the 1st patch. Otherwise the patch history will be a bit messy.

src/cephadm/cephadm

adk3798

LGTM

Fixes: https://tracker.ceph.com/issues/55008 Fixes: https://tracker.ceph.com/issues/54373 Signed-off-by: Redouane Kachach <rkachach@redhat.com>

src/cephadm/cephadm

…podman Signed-off-by: Redouane Kachach <rkachach@redhat.com>

rkachach · 2022-04-26T05:54:43Z

jenkins test make check

adk3798 · 2022-04-26T15:44:47Z

https://pulpito.ceph.com/adking-2022-04-23_01:33:16-orch:cephadm-wip-adk-testing-2022-04-22-1640-distro-basic-smithi/

2 SELinux Denial Failures Tracked by https://tracker.ceph.com/issues/55443

rkachach added the cephadm label Mar 23, 2022

rkachach requested a review from a team as a code owner March 23, 2022 17:31

rkachach force-pushed the fix_issue_55008 branch 2 times, most recently from d67424c to 97fbab6 Compare March 28, 2022 14:19

adk3798 added the wip-adk-testing label Mar 31, 2022

mgfritch reviewed Apr 13, 2022

View reviewed changes

phlogistonjohn approved these changes Apr 18, 2022

View reviewed changes

rkachach force-pushed the fix_issue_55008 branch from 97fbab6 to 5d91887 Compare April 19, 2022 16:06

rkachach requested review from phlogistonjohn and mgfritch April 19, 2022 16:07

rkachach force-pushed the fix_issue_55008 branch 3 times, most recently from c036603 to f4c021d Compare April 19, 2022 16:50

adk3798 reviewed Apr 19, 2022

View reviewed changes

rkachach requested a review from adk3798 April 20, 2022 10:00

adk3798 reviewed Apr 20, 2022

View reviewed changes

phlogistonjohn requested changes Apr 20, 2022

View reviewed changes

rkachach force-pushed the fix_issue_55008 branch from bd8786a to 5d6ef23 Compare April 20, 2022 15:31

rkachach requested review from phlogistonjohn and adk3798 April 20, 2022 15:31

adk3798 reviewed Apr 21, 2022

View reviewed changes

src/cephadm/cephadm Outdated Show resolved Hide resolved

src/cephadm/cephadm Outdated Show resolved Hide resolved

adk3798 approved these changes Apr 21, 2022

View reviewed changes

mgr/cephadm: Adding image tag and date to cephadm startup messages

92ecb58

Fixes: https://tracker.ceph.com/issues/55008 Fixes: https://tracker.ceph.com/issues/54373 Signed-off-by: Redouane Kachach <rkachach@redhat.com>

rkachach force-pushed the fix_issue_55008 branch from 5d6ef23 to 92ecb58 Compare April 21, 2022 07:50

adk3798 requested changes Apr 22, 2022

View reviewed changes

src/cephadm/cephadm Outdated Show resolved Hide resolved

mgr/cephadm: Using '.ID' instead of '.Id' which is only supported by …

6e7c048

…podman Signed-off-by: Redouane Kachach <rkachach@redhat.com>

adk3798 approved these changes Apr 26, 2022

View reviewed changes

phlogistonjohn approved these changes Apr 26, 2022

View reviewed changes

adk3798 merged commit 11fde33 into ceph:master Apr 26, 2022
12 checks passed

mgr/cephadm: Adding image tag and date to cephadm startup messages #45598

mgr/cephadm: Adding image tag and date to cephadm startup messages #45598

rkachach commented Mar 23, 2022 •

edited

rkachach commented Mar 29, 2022

adk3798 commented Apr 4, 2022

rkachach commented Apr 5, 2022

adk3798 commented Apr 11, 2022

adk3798 commented Apr 13, 2022

mgfritch Apr 13, 2022

rkachach Apr 19, 2022

adk3798 commented Apr 15, 2022

phlogistonjohn left a comment

adk3798 left a comment

phlogistonjohn left a comment

adk3798 left a comment

rkachach commented Apr 26, 2022

adk3798 commented Apr 26, 2022

mgr/cephadm: Adding image tag and date to cephadm startup messages #45598

mgr/cephadm: Adding image tag and date to cephadm startup messages #45598

Conversation

rkachach commented Mar 23, 2022 • edited

Checklist

rkachach commented Mar 29, 2022

adk3798 commented Apr 4, 2022

rkachach commented Apr 5, 2022

adk3798 commented Apr 11, 2022

adk3798 commented Apr 13, 2022

mgfritch Apr 13, 2022

Choose a reason for hiding this comment

rkachach Apr 19, 2022

Choose a reason for hiding this comment

adk3798 commented Apr 15, 2022

phlogistonjohn left a comment

Choose a reason for hiding this comment

adk3798 left a comment

Choose a reason for hiding this comment

phlogistonjohn left a comment

Choose a reason for hiding this comment

adk3798 left a comment

Choose a reason for hiding this comment

rkachach commented Apr 26, 2022

adk3798 commented Apr 26, 2022

rkachach commented Mar 23, 2022 •

edited