Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

quincy: osd, tools, kv: non-aggressive, on-line trimming of accumulated dups #47688

Merged
merged 7 commits into from Aug 26, 2022
Merged

quincy: osd, tools, kv: non-aggressive, on-line trimming of accumulated dups #47688

merged 7 commits into from Aug 26, 2022

Conversation

rzarzynski
Copy link
Contributor

This is a quincy backport of #47046.

Contribution Guidelines

Checklist

  • Tracker (select at least one)
    • References tracker ticket
    • Very recent bug; references commit where it was introduced
    • New feature (ticket optional)
    • Doc update (no ticket needed)
    • Code cleanup (no ticket needed)
  • Component impact
    • Affects Dashboard, opened tracker ticket
    • Affects Orchestrator, opened tracker ticket
    • No impact that needs to be tracked
  • Documentation (select at least one)
    • Updates relevant documentation
    • No doc update is appropriate
  • Tests (select at least one)
Show available Jenkins commands
  • jenkins retest this please
  • jenkins test classic perf
  • jenkins test crimson perf
  • jenkins test signed
  • jenkins test make check
  • jenkins test make check arm64
  • jenkins test submodules
  • jenkins test dashboard
  • jenkins test dashboard cephadm
  • jenkins test api
  • jenkins test docs
  • jenkins render docs
  • jenkins test ceph-volume all
  • jenkins test ceph-volume tox
  • jenkins test windows

NitzanMordhai and others added 7 commits August 18, 2022 21:42
@NitzanMordhai @rzarzynski
tools/ceph_objectstore_tool: Add duplicate pg log entries injection
124d097 
To be able to recreate and test pg log duplicate entries, a new option
added to the COT: --op pg-log-inject-dups we will also need to provide
--file json_arry of dups, it can get as many dups that need to be inject

the json for dups is in the following format:
{"reqid": "client.n.n:n", "version": "n'n", "user_version": n, "return_code": n}

Signed-off-by: Nitzan Mordechai <nmordec@redhat.com>
(cherry picked from commit b59ccfd)
@rzarzynski
osd: trim dups regardless of their versions
9461ab0 
Fixes: https://tracker.ceph.com/issues/53729
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
(cherry picked from commit aada08a)
@NitzanMordhai @rzarzynski
test/PGlog: cherry-pick PGLogTrimTest changes for size-based trimming
b67e852 
This commit brings the changes to the `PGLogTrimTest` that were a part
of the initial fix for dups trimming issue (commit 0d253bc)
that got reverted in the meantime by 9bf0053.

Signed-off-by: Nitzan Mordechai <nmordec@redhat.com>
(cherry picked from commit 245fa48)
@rzarzynski
osd, tools: read_log_and_missing of PGLog takes CephContext
145b331 
Will be needed for dealing with the conf.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
(cherry picked from commit fb746dd)
@rzarzynski
osd: warn about too many dups when loading PGLog
f2cdf5b 
Fixes: https://tracker.ceph.com/issues/53729
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
(cherry picked from commit 7376caa)
@rzarzynski
osd: write_log_and_missing variants of PGLog take DoutPrefixProvider
623014d 
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
(cherry picked from commit ff8dccd)
@rzarzynski
osd, kv: add debugs along the pg_log_t::dups paths.
b06c8ff 
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
(cherry picked from commit 0a986bf)
@rzarzynski rzarzynski added this to the quincy milestone Aug 18, 2022
@rzarzynski rzarzynski requested a review from a team as a code owner August 18, 2022 21:46
@rzarzynski
Copy link
Contributor Author

jenkins retest this please

@rzarzynski
Copy link
Contributor Author

jenkins test make check

rzarzynski added a commit to rzarzynski/ceph that referenced this pull request Aug 23, 2022
@rzarzynski
PendingReleaseNotes: document online and offline trimming of PG Log's…
015d2bd 
… dups

This commit aggregates changes for multiple PR:

* Offline: ceph#46630
* Online: ceph#47046

* Offline fix: ceph#46706
* Online fix: ceph#47688

* Offline fix: ceph#46631
* Online fix: ceph#47701

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
rzarzynski added a commit to rzarzynski/ceph that referenced this pull request Aug 23, 2022
@rzarzynski
PendingReleaseNotes: document online and offline trimming of PG Log's…
560e9d8 
… dups

This commit aggregates changes for multiple PR:

* Offline: ceph#46630
* Online: ceph#47046

* Offline fix: ceph#46706
* Online fix: ceph#47688

* Offline fix: ceph#46631
* Online fix: ceph#47701

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
rzarzynski added a commit to rzarzynski/ceph that referenced this pull request Aug 23, 2022
@rzarzynski
PendingReleaseNotes: document online and offline trimming of PG Log's…
e83387c 
… dups

This commit aggregates changes for multiple PR:

main
----
* Offline: ceph#46630
* Online: ceph#47046

quincy
------
* Offline fix: ceph#46706
* Online fix: ceph#47688

pacific
-------
* Offline fix: ceph#46631
* Online fix: ceph#47701

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
@rzarzynski rzarzynski mentioned this pull request Aug 23, 2022
Merged
14 tasks
rzarzynski added a commit to rzarzynski/ceph that referenced this pull request Aug 23, 2022
@rzarzynski
PendingReleaseNotes: document online and offline trimming of PG Log's…
7c557e4 
… dups

This commit aggregates changes for multiple PR:

main
----
* Offline: ceph#46630
* Online: ceph#47046

quincy
------
* Offline fix: ceph#46706
* Online fix: ceph#47688

pacific
-------
* Offline fix: ceph#46631
* Online fix: ceph#47701

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
@ljflores
Copy link
Contributor

Rados suite review: https://pulpito.ceph.com/?branch=wip-yuri3-testing-2022-08-24-0820-quincy

Failures, unrelated:
1. https://tracker.ceph.com/issues/57303 -- Pending Quincy backport
2. https://tracker.ceph.com/issues/57269
3. https://tracker.ceph.com/issues/57165 -- Fix under review
4. https://tracker.ceph.com/issues/57270 -- Pending Quincy backport
5. https://tracker.ceph.com/issues/57311

Details:
1. rados/cephadm: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b34ca7d1c2becd6090874ccda56ef4cd8dc64bf7 - Ceph - Orchestrator
2. rook: unable to read URL "https://docs.projectcalico.org/manifests/tigera-operator.yaml" - Ceph - Orchestrator
3. expected valgrind issues and found none - Ceph - RADOS
4. cephadm: RuntimeError: Failed command: apt-get update: E: The repository 'https://download.ceph.com/debian-octopus jammy Release' does not have a Release file. - Ceph - Orchestrator
5. rook: ensure CRDs are installed first - Ceph - Orchestrator

@ljflores ljflores merged commit 3c61722 into ceph:quincy Aug 26, 2022
9 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ljflores ljflores ljflores approved these changes

@neha-ojha neha-ojha Awaiting requested review from neha-ojha

Assignees
No one assigned
Labels
core needs-qa needs-release-note tests wip-yuri3-testing
Projects
None yet
Milestone
quincy
5 participants
@rzarzynski @ljflores @yuriw @neha-ojha @NitzanMordhai