DAOS-11936 object: fix online extending bugs by wangdi1 · Pull Request #10604 · daos-stack/daos

wangdi1 · 2022-10-19T00:04:08Z

Only checking non-extending shards in obj_grp_valid_shard_get().
Recreate the bulk during retry, in case the bulk needs to be created and re-binded, once refreshing the layout.

Required-githooks: true

Signed-off-by: Di Wang di.wang@intel.com

Before requesting gatekeeper:

Two review approvals and any prior change requests have been resolved.
Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
Commit messages follows the guidelines outlined here.
Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

github-actions · 2022-10-19T00:04:33Z

Bug-tracker data:
Ticket title is 'ERROR: dfs_write(0x5591d7e8e000, 40000) failed (22): Invalid argument. during online extend test.'
Status is 'In Review'
Job should run at elevated priority (3)
https://daosio.atlassian.net/browse/DAOS-11936

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2022-10-19T15:27:39Z

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-10604/1/execution/node/1083/log

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2022-10-20T18:37:46Z

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-10604/2/execution/node/1130/log

daosbuild1

LGTM. No errors found by checkpatch.

liuxuezhao · 2022-10-21T06:45:15Z

src/object/cli_obj.c

+	 * So let's free the existent bulk, and recreate the bulk later.
+	 */
+	if (obj_auxi->io_retry && obj_auxi->bulks != NULL)
+		obj_bulk_fini(obj_auxi);


just confirm for this bulk change, why need to recreate the bulk handle?
note that for crt_bulk_bind() is it to bind local context address to the bulk handle (not remote target address), so not sure why need to recreate (for rebind?) here?

yes, this is for rebinding. for osa online case. it is possible that the initial I/O will not need bind the bulk because it does not forward, but retry will refresh the pool map, then need forward (need bulk the bind).

On the other hand, the initial I/O might need forward, but then retry might not need forward.

So we might just delete the bulk and recreate and re-bind the bulk as needed.

liuxuezhao · 2022-10-21T06:48:13Z

src/placement/pl_map_common.c

 			new_shards[grp_idx].po_rebuilding = 1;

-		if (f_shard->fs_status == PO_COMP_ST_UP)
+		if (f_shard->fs_status == PO_COMP_ST_UP || f_shard->fs_status == PO_COMP_ST_NEW)


just confirm, a reintegrating tgt, it should be in PO_COMP_ST_UP state? (see update_one_tgt, case POOL_REINT)
Is it possible to get the NEW shard here?

oh, it could be NEW for extending targets.

daltonbohning · 2022-10-21T16:27:41Z

src/tests/ftest/osa/online_extend.py

+            if app_name == "ior":
+                self.run_ior_thread("Read", oclass, test_seq)


I'm not familiar with this test, but do we need something like this?

else: self.run_mdtest_thread()

oh, I think that is for verification as I thought. I am not sure if there are separate verification for mdtest? @rpadma2

Presently, mdtest doesn't do separate write/read. It just runs mdtest on the same container read/write together. That's all... In future, we can enhance it (if needed). IOR is only used as separate write and read now.

We should run mdtest with read & stat to verify the tree. Let's create another ticket for this.

daosbuild1 · 2022-10-21T20:53:00Z

Test stage Functional Hardware Large completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-10604/3/testReport/(root)/

daosbuild1 · 2022-10-21T21:51:56Z

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-10604/3/execution/node/1083/log

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2022-10-23T00:25:05Z

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-10604/4/execution/node/915/log

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2022-10-24T14:06:24Z

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-10604/6/execution/node/916/log

1. Only checking non-extending shards in obj_grp_valid_shard_get(). 2. Recreate the bulk during retry, in case the bulk needs to be created and re-binded, once refreshing the layout. 3. allow_version only be valid if it is > 0. 4. EXTEND target should also be reintegrating status. 5. Set pool rebuild version for the client between servers. Required-githooks: true Signed-off-by: Di Wang <di.wang@intel.com>

daosbuild1

LGTM. No errors found by checkpatch.

johannlombardi · 2022-10-25T14:30:49Z

src/tests/ftest/osa/online_extend.py

+            if app_name == "ior":
+                self.run_ior_thread("Read", oclass, test_seq)


We should run mdtest with read & stat to verify the tree. Let's create another ticket for this.

wangdi1 requested review from jolivier23 and liuxuezhao October 19, 2022 00:04

github-actions bot added priority Ticket has high priority (automatically managed) release-2.4 PR is eventually targeted for 2.4 labels Oct 19, 2022

daosbuild1 reviewed Oct 19, 2022

View reviewed changes

bfaccini mentioned this pull request Oct 19, 2022

DAOS-11149 pool: bump GC ULT stack size #9924

Merged

wangdi1 force-pushed the wangdi/daos_11936 branch from 71ac44f to c71b561 Compare October 19, 2022 23:48

daosbuild1 reviewed Oct 19, 2022

View reviewed changes

jolivier23 approved these changes Oct 20, 2022

View reviewed changes

jolivier23 changed the title ~~DAOS-11936 objects: fix online extending bugs~~ DAOS-11936 object: fix online extending bugs Oct 20, 2022

jolivier23 previously approved these changes Oct 20, 2022

View reviewed changes

wangdi1 dismissed jolivier23’s stale review via c13bd16 October 21, 2022 01:22

wangdi1 force-pushed the wangdi/daos_11936 branch from c71b561 to c13bd16 Compare October 21, 2022 01:22

wangdi1 requested a review from a team as a code owner October 21, 2022 01:22

daosbuild1 reviewed Oct 21, 2022

View reviewed changes

liuxuezhao reviewed Oct 21, 2022

View reviewed changes

daltonbohning reviewed Oct 21, 2022

View reviewed changes

wangdi1 force-pushed the wangdi/daos_11936 branch from c13bd16 to fbeb466 Compare October 21, 2022 22:31

daosbuild1 reviewed Oct 21, 2022

View reviewed changes

wangdi1 force-pushed the wangdi/daos_11936 branch from fbeb466 to 220316b Compare October 23, 2022 05:48

daosbuild1 reviewed Oct 23, 2022

View reviewed changes

wangdi1 force-pushed the wangdi/daos_11936 branch from 220316b to 3a0f9b2 Compare October 24, 2022 16:17

daosbuild1 reviewed Oct 24, 2022

View reviewed changes

wangdi1 requested review from daltonbohning, jolivier23 and liuxuezhao October 24, 2022 17:15

jolivier23 approved these changes Oct 24, 2022

View reviewed changes

johannlombardi approved these changes Oct 25, 2022

View reviewed changes

johannlombardi merged commit 431340f into master Oct 25, 2022

johannlombardi deleted the wangdi/daos_11936 branch October 25, 2022 14:31

		if app_name == "ior":
		self.run_ior_thread("Read", oclass, test_seq)

Conversation

wangdi1 commented Oct 19, 2022

Before requesting gatekeeper:

Gatekeeper:

Uh oh!

github-actions bot commented Oct 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daosbuild1 left a comment

Choose a reason for hiding this comment

Uh oh!

daosbuild1 commented Oct 19, 2022

Uh oh!

daosbuild1 left a comment

Choose a reason for hiding this comment

Uh oh!

daosbuild1 commented Oct 20, 2022

Uh oh!

daosbuild1 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daosbuild1 commented Oct 21, 2022

Uh oh!

daosbuild1 commented Oct 21, 2022

Uh oh!

daosbuild1 left a comment

Choose a reason for hiding this comment

Uh oh!

daosbuild1 commented Oct 23, 2022

Uh oh!

daosbuild1 left a comment

Choose a reason for hiding this comment

Uh oh!

daosbuild1 commented Oct 24, 2022

Uh oh!

daosbuild1 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

7 participants

github-actions bot commented Oct 19, 2022 •

edited

Loading