pacific: os/bluestore: fix spillover alert #50932

ifed01 · 2023-04-07T10:06:15Z

backport of #49987

backport tracker: https://tracker.ceph.com/issues/59340
parent tracker: https://tracker.ceph.com/issues/58440

Signed-off-by: Igor Fedotov igor.fedotov@croit.io

Contribution Guidelines

To sign and title your commits, please refer to Submitting Patches to Ceph.
If you are submitting a fix for a stable branch (e.g. "pacific"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

Checklist

Tracker (select at least one)
- References tracker ticket
- Very recent bug; references commit where it was introduced
- New feature (ticket optional)
- Doc update (no ticket needed)
- Code cleanup (no ticket needed)
Component impact
- Affects Dashboard, opened tracker ticket
- Affects Orchestrator, opened tracker ticket
- No impact that needs to be tracked
Documentation (select at least one)
- Updates relevant documentation
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)
- Includes integration test(s)
- Includes bug reproducer
- No tests

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard cephadm
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox
jenkins test windows

Signed-off-by: Igor Fedotov <igor.fedotov@croit.io> (cherry picked from commit 431ca85)

Fixes: https://tracker.ceph.com/issues/58440 Signed-off-by: Igor Fedotov <igor.fedotov@croit.io> (cherry picked from commit 4f20521)

Signed-off-by: Igor Fedotov <igor.fedotov@croit.io> (cherry picked from commit 326eabe)

ljflores · 2023-05-10T23:47:45Z

Hey @ifed01, I'm seeing a bluestore test fail consistently in the QA runs. Can you take a look?

http://pulpito.front.sepia.ceph.com/?branch=wip-yuri11-testing-2023-04-25-1605-pacific

/a/yuriw-2023-04-26_01:16:19-rados-wip-yuri11-testing-2023-04-25-1605-pacific-distro-default-smithi/7253708
/a/yuriw-2023-05-01_16:31:04-rados-wip-yuri11-testing-2023-04-25-1605-pacific-distro-default-smithi/7259813
/a/lflores-2023-05-03_21:43:07-rados-wip-yuri11-testing-2023-04-25-1605-pacific-distro-default-smithi/7261653

2023-04-29T06:18:33.927 INFO:teuthology.orchestra.run.smithi155.stdout:==> rm -r bluestore.test_temp_dir
2023-04-29T06:18:33.931 INFO:teuthology.orchestra.run.smithi155.stdout:[       OK ] ObjectStore/StoreTest.MultipoolListTest/2 (3440 ms)
2023-04-29T06:18:33.931 INFO:teuthology.orchestra.run.smithi155.stdout:[ RUN      ] ObjectStore/StoreTest.SimpleCloneTest/0
2023-04-29T06:18:34.015 INFO:teuthology.orchestra.run.smithi155.stderr:Creating collection meta
2023-04-29T06:18:34.015 INFO:teuthology.orchestra.run.smithi155.stderr:Creating object and set attr #-1:de000000::key:Object 1:head#
2023-04-29T06:18:34.016 INFO:teuthology.orchestra.run.smithi155.stderr:Clone object and rm attr
2023-04-29T06:18:34.016 INFO:teuthology.orchestra.run.smithi155.stderr:Invalid rm coll
2023-04-29T06:18:34.055 INFO:teuthology.orchestra.run.smithi155.stderr:Invalid rm coll again
2023-04-29T06:18:34.095 INFO:teuthology.orchestra.run.smithi155.stderr:Cleaning
2023-04-29T06:18:34.095 INFO:teuthology.orchestra.run.smithi155.stdout:==> rm -r memstore.test_temp_dir
2023-04-29T06:18:34.097 INFO:teuthology.orchestra.run.smithi155.stdout:[       OK ] ObjectStore/StoreTest.SimpleCloneTest/0 (167 ms)
2023-04-29T06:18:34.097 INFO:teuthology.orchestra.run.smithi155.stdout:[ RUN      ] ObjectStore/StoreTest.SimpleCloneTest/1
2023-04-29T06:18:34.856 INFO:teuthology.orchestra.run.smithi155.stderr:Creating collection meta
2023-04-29T06:18:34.856 INFO:teuthology.orchestra.run.smithi155.stderr:Creating object and set attr #-1:de000000::key:Object 1:head#
2023-04-29T06:18:34.856 INFO:teuthology.orchestra.run.smithi155.stderr:Clone object and rm attr
2023-04-29T06:18:35.976 INFO:teuthology.orchestra.run.smithi155.stderr:Invalid rm coll
2023-04-29T06:18:36.013 INFO:teuthology.orchestra.run.smithi155.stderr:Invalid rm coll again
2023-04-29T06:18:36.558 INFO:teuthology.orchestra.run.smithi155.stderr:Cleaning
2023-04-29T06:18:36.837 INFO:teuthology.orchestra.run.smithi155.stdout:==> rm -r filestore.test_temp_dir
2023-04-29T06:18:36.840 INFO:teuthology.orchestra.run.smithi155.stdout:[       OK ] ObjectStore/StoreTest.SimpleCloneTest/1 (2743 ms)
2023-04-29T06:18:36.841 INFO:teuthology.orchestra.run.smithi155.stdout:[ RUN      ] ObjectStore/StoreTest.SimpleCloneTest/2
2023-04-29T06:18:40.072 INFO:teuthology.orchestra.run.smithi155.stderr:Creating collection meta
2023-04-29T06:18:40.073 INFO:teuthology.orchestra.run.smithi155.stderr:Creating object and set attr #-1:de000000::key:Object 1:head#
2023-04-29T06:18:40.073 INFO:teuthology.orchestra.run.smithi155.stderr:Clone object and rm attr
2023-04-29T06:18:41.668 INFO:teuthology.orchestra.run.smithi155.stderr:Invalid rm coll
2023-04-29T18:06:12.144 DEBUG:teuthology.exit:Got signal 15; running 1 handler...
2023-04-29T18:06:12.253 DEBUG:teuthology.task.console_log:Killing console logger for smithi155
2023-04-29T18:06:12.254 DEBUG:teuthology.exit:Finished running handlers

/a/yuriw-2023-04-26_01:16:19-rados-wip-yuri11-testing-2023-04-25-1605-pacific-distro-default-smithi/7253708/remote/smithi155/ceph_test_objectstore.log.gz

2023-04-29T06:18:34.015+0000 7f1bbeb84540  0 memstore(memstore.test_temp_dir) dump:{
    "collections": [
        {
            "name": "meta",
            "xattrs": [],
            "objects": [
                {
                    "name": "#-1:de000000::key:Object 1:head#",
                    "data_len": 65536,
                    "omap_header_len": 0,
                    "xattrs": [],
                    "omap": []
                },
                {
                    "name": "#-1:de000000::key:Object 2:head#",
                    "data_len": 65536,
                    "omap_header_len": 0,
                    "xattrs": [],
                    "omap": []
                }
            ]
        }
    ]
}

2023-04-29T06:18:34.023+0000 7f1bbeb84540 -1 *** Caught signal (Aborted) **
 in thread 7f1bbeb84540 thread_name:ceph_test_objec

 ceph version 16.2.12-89-g8d175760 (8d17576050e846ecd4a9899bc7d8ebbf771b4de8) pacific (stable)
 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f1bc8546420]
 2: gsignal()
 3: abort()
 4: ceph_test_objectstore(+0x80f89b) [0x5609e1eaf89b]
 5: ceph_test_objectstore(+0xc06176) [0x5609e22a6176]
 6: clone()

ifed01 · 2023-05-11T09:19:34Z

Hey @ifed01, I'm seeing a bluestore test fail consistently in the QA runs. Can you take a look?

Hey @ljflores - this looks absolutely similar to the error you shared at #50506 two weeks ago.
And alike to that case it looks rather irrelevant since it's memstore which is failing. While the PRs in question don't deal with this specific object a]store. So I'm curious if there are any common PRs/commits which were included in both QA runs? If not I would say the issue is caused by a completely different stuff, e.g. prior commits or something...

ljflores · 2023-05-15T19:33:48Z

Hey @ifed01, I'm seeing a bluestore test fail consistently in the QA runs. Can you take a look?

Hey @ljflores - this looks absolutely similar to the error you shared at #50506 two weeks ago. And alike to that case it looks rather irrelevant since it's memstore which is failing. While the PRs in question don't deal with this specific object a]store. So I'm curious if there are any common PRs/commits which were included in both QA runs? If not I would say the issue is caused by a completely different stuff, e.g. prior commits or something...

@ifed01 Here was the Trello card this was tested on for reference: https://trello.com/c/RFvB8Ugn/1741-wip-yuri11-testing-2023-04-25-1605-pacific

See #50506 (comment) for how I think we should handle this.

ljflores · 2023-05-16T15:46:53Z

Rados suite review available here: https://tracker.ceph.com/projects/rados/wiki/PACIFIC#httpstrellocomcRFvB8Ugn1741-wip-yuri11-testing-2023-04-25-1605-pacific

ifed01 added 3 commits April 7, 2023 13:04

test/store_test: BlueFSSpilloverTest to check corresponding alert

89e3bba

Signed-off-by: Igor Fedotov <igor.fedotov@croit.io> (cherry picked from commit 431ca85)

os/bluestore: bring spillover alerts back after regression.

5b2eb37

Fixes: https://tracker.ceph.com/issues/58440 Signed-off-by: Igor Fedotov <igor.fedotov@croit.io> (cherry picked from commit 4f20521)

os/bluestore: beautify disk size output for volume selector

4bbfea8

Signed-off-by: Igor Fedotov <igor.fedotov@croit.io> (cherry picked from commit 326eabe)

ifed01 requested a review from a team as a code owner April 7, 2023 10:06

ifed01 requested a review from aclamk April 7, 2023 10:06

ifed01 added bluestore bug-fix backport labels Apr 7, 2023

github-actions bot added core tests labels Apr 7, 2023

github-actions bot added this to the pacific milestone Apr 7, 2023

yuriw added needs-qa wip-yuri11-testing labels Apr 25, 2023

yuriw removed the wip-yuri11-testing label May 11, 2023

ljflores mentioned this pull request May 15, 2023

pacific: os/bluestore: report min_alloc_size through "ceph osd metadata" #50506

Merged

14 tasks

ljflores added the wip-lflores-testing label May 15, 2023

ljflores approved these changes May 16, 2023

View reviewed changes

yuriw merged commit 7ec3217 into ceph:pacific May 16, 2023
8 checks passed

ifed01 deleted the wip-ifed-fix-spillover-alert-pac branch May 16, 2023 15:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pacific: os/bluestore: fix spillover alert #50932

pacific: os/bluestore: fix spillover alert #50932

ifed01 commented Apr 7, 2023

ljflores commented May 10, 2023 •

edited

ifed01 commented May 11, 2023 •

edited

ljflores commented May 15, 2023 •

edited

ljflores commented May 16, 2023

pacific: os/bluestore: fix spillover alert #50932

pacific: os/bluestore: fix spillover alert #50932

Conversation

ifed01 commented Apr 7, 2023

Contribution Guidelines

Checklist

ljflores commented May 10, 2023 • edited

ifed01 commented May 11, 2023 • edited

ljflores commented May 15, 2023 • edited

ljflores commented May 16, 2023

ljflores commented May 10, 2023 •

edited

ifed01 commented May 11, 2023 •

edited

ljflores commented May 15, 2023 •

edited