HBASE-28001: Add request attribute support to BufferedMutator #6076

eab148 · 2024-07-12T13:20:05Z

As of #5326, we have been able to set request attributes on Table instances via TableBuilder::setRequestAttribute. With request attributes, users can send an attribute once per request, instead of once per operation.

In this PR, we extend the feature to BufferedMutators. One can set a request attribute on a BufferedMutator via a new method BufferedMutatorParams::setRequestAttribute(String key, byte[] value).

Map<String, byte[]> REQUEST_ATTRIBUTES = Map.of(...);
...
BufferedMutatorParams params = 
    new BufferedMutatorParams(REQUEST_ATTRIBUTES_TEST_TABLE);
REQUEST_ATTRIBUTES.forEach(params::setRequestAttribute);
BufferedMutator bufferedMutator = conn.getBufferedMutator(params);
// all calls generated via this buffered mutator will include the given attributes

Jira

cc @bbeaudreault @hgromer @bozzkar @rmdmattingly @krconv @sidkhillon

ndimiduk

No objections from me. This looks like a straightforward extension of our existing feature. It's been a while since I've thought about these classes, but the use-case and API changes look fine to me.

Since this is public API, let's see if others have an opinion here.

hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncBufferedMutator.java

ndimiduk · 2024-07-17T08:45:08Z

hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncBufferedMutatorBuilder.java

+  /**
+   * Set a rpc request attribute.
+   */
+  AsyncBufferedMutatorBuilder setRequestAttribute(String key, byte[] value);


Do we expect this to override all existing requestAttributes and replace the collection with this value? Or, do we expect this method to add an additional requestAttribute to an existing set?

Take a look at the breadth of API exposed on Immutable.Builder implementations around collection objects -- https://immutables.github.io/immutable.html#array-collection-and-map-attributes I'm not saying that we need all of these, but at least consider which semantics we want to support and why.

Ah, or maybe you're hamstrung by the apis that TableBuilder exposes? Maybe we should look at expanding the scope of these methods in a separate JIRA.

Ah, or maybe you're hamstrung by the apis that TableBuilder exposes?

Yeah, currently the AsyncBufferedMutatorBuilderImpl wraps an AsyncTableBuilder (code).

Maybe we should look at expanding the scope of these methods in a separate JIRA.

I think this is a good idea. When I drafted this PR, I had a difficult time understanding what semantics we support and why we chose them.

hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncTableImpl.java

...-server/src/test/java/org/apache/hadoop/hbase/client/TestRequestAndConnectionAttributes.java

ndimiduk · 2024-07-17T09:02:43Z

Do we have anything that demonstrates use of connection or request attributes in hbase-examples? If yes, please add example overage there. If not, would you mind filing an issue to add something to that module? Thanks @eab148 !

eab148 · 2024-07-17T18:43:44Z

Do we have anything that demonstrates use of connection or request attributes in hbase-examples? If yes, please add example overage there. If not, would you mind filing an issue to add something to that module?

We do not. I can create an issue to add request attribute example coverage to the hbase-examples module.

eab148 · 2024-07-17T18:44:31Z

Since this is public API, let's see if others have an opinion here.

Makes sense! Happy to field any questions/concerns and make adjustments.

eab148 · 2024-08-23T11:46:39Z

Test failure is unrelated to this change

[ERROR] Tests run: 7, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 768.803 s <<< FAILURE! - in org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy
[ERROR] org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy  Time elapsed: 618.862 s  <<< ERROR!
org.junit.runners.model.TestTimedOutException: test timed out after 780 seconds
	at java.base@17.0.11/java.lang.Object.wait(Native Method)
	at java.base@17.0.11/java.lang.Thread.join(Thread.java:1313)
	at java.base@17.0.11/java.lang.Thread.join(Thread.java:1381)
	at app//org.apache.hadoop.hbase.MultithreadedTestUtil$TestContext.stop(MultithreadedTestUtil.java:103)
	at app//org.apache.hadoop.hbase.AcidGuaranteesTestTool.runTestAtomicity(AcidGuaranteesTestTool.java:390)
	at app//org.apache.hadoop.hbase.AcidGuaranteesTestTool.doWork(AcidGuaranteesTestTool.java:130)
	at app//org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:150)
	at app//org.apache.hadoop.hbase.AcidGuaranteesTestBase.runTestAtomicity(AcidGuaranteesTestBase.java:102)
	at app//org.apache.hadoop.hbase.AcidGuaranteesTestBase.testMobGetAtomicity(AcidGuaranteesTestBase.java:122)
	at java.base@17.0.11/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base@17.0.11/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base@17.0.11/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base@17.0.11/java.lang.reflect.Method.invoke(Method.java:568)
	at app//org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at app//org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at app//org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at app//org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at app//org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at app//org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at app//org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at app//org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at app//org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at app//org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at app//org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at app//org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at app//org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at app//org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at app//org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at app//org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at app//org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at app//org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at app//org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	at app//org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	at java.base@17.0.11/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base@17.0.11/java.lang.Thread.run(Thread.java:840)

[ERROR] org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy  Time elapsed: 618.878 s  <<< ERROR!
java.lang.Exception: Appears to be stuck in thread MiniHBaseClusterRegionServer-EventLoopGroup-3-1
	at app//org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method)
	at app//org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:193)
	at app//org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:312)
	at app//org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:376)
	at app//org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at app//org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at app//org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base@17.0.11/java.lang.Thread.run(Thread.java:840)

Apache-HBase · 2024-09-05T09:04:31Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 44s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ master Compile Tests _
+0 🆗	mvndep	0m 35s		Maven dependency ordering for branch
+1 💚	mvninstall	3m 29s		master passed
+1 💚	compile	4m 15s		master passed
+1 💚	checkstyle	0m 59s		master passed
+1 💚	spotbugs	2m 46s		master passed
+1 💚	spotless	0m 45s		branch has no errors when running spotless:check.
-0 ⚠️	patch	1m 3s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 10s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 2s		the patch passed
+1 💚	compile	4m 13s		the patch passed
+1 💚	javac	4m 13s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 59s		the patch passed
+1 💚	spotbugs	3m 3s		the patch passed
+1 💚	hadoopcheck	11m 35s		Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚	spotless	0m 48s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 28s		The patch does not generate ASF License warnings.
		45m 27s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6076/10/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#6076
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname	Linux 6a199395ebc9 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `0a04b60`
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	84 (vs. ulimit of 30000)
modules	C: hbase-client hbase-server hbase-examples U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6076/10/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-09-05T12:37:09Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 34s		Docker mode activated.
-0 ⚠️	yetus	0m 3s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ master Compile Tests _
+0 🆗	mvndep	0m 18s		Maven dependency ordering for branch
+1 💚	mvninstall	3m 5s		master passed
+1 💚	compile	1m 43s		master passed
+1 💚	javadoc	1m 7s		master passed
+1 💚	shadedjars	5m 17s		branch has no errors when building our shaded downstream artifacts.
-0 ⚠️	patch	5m 45s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 13s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 57s		the patch passed
+1 💚	compile	1m 42s		the patch passed
+1 💚	javac	1m 42s		the patch passed
+1 💚	javadoc	1m 6s		the patch passed
+1 💚	shadedjars	5m 20s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	1m 37s		hbase-client in the patch passed.
+1 💚	unit	225m 49s		hbase-server in the patch passed.
+1 💚	unit	2m 13s		hbase-examples in the patch passed.
		258m 5s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6076/10/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#6076
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux 3f5fc70dab73 5.4.0-192-generic #212-Ubuntu SMP Fri Jul 5 09:47:39 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `0a04b60`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6076/10/testReport/
Max. process+thread count	5142 (vs. ulimit of 30000)
modules	C: hbase-client hbase-server hbase-examples U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6076/10/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

…#6076) Co-authored-by: Evie Boland <eboland@hubspot.com> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>

ndimiduk · 2024-09-05T14:01:59Z

Thanks a lot for the contribution @eab148 ! Can you please also provide a new PR for branch-2? You can start by cherry-picking the commit that landed on master. Do ping me when you have the PR up, thanks!

Co-authored-by: Evie Boland <eboland@hubspot.com> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>

…#6076) Co-authored-by: Evie Boland <eboland@hubspot.com> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>

Co-authored-by: Evie Boland <eboland@hubspot.com> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>

…edMutator (apache#6076) Co-authored-by: Evie Boland <eboland@hubspot.com> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>

…redMutator (apache#6076) Co-authored-by: Evie Boland <eboland@hubspot.com> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>

…edMutator (apache#6076) (#116) Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Co-authored-by: Evie Boland <eboland@hubspot.com>

…redMutator (apache#6076) (#118) Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Co-authored-by: Evie Boland <eboland@hubspot.com>

…apache#6076)" This reverts commit 7d019d9.

…#6076) (addendum) Add default implementations of the new methods so that a custom implementation of AsyncBufferedMutatorBuilder will not fail to compile after upgrade.

… (addendum) (#6349) Add default implementations of the new methods so that a custom implementation of AsyncBufferedMutatorBuilder will not fail to compile after upgrade. Signed-off-by: Nihal Jain <nihaljain@apache.org> Signed-off-by: Istvan Toth <stoty@apache.org>

…#6076) (addendum) (apache#6349) Add default implementations of the new methods so that a custom implementation of AsyncBufferedMutatorBuilder will not fail to compile after upgrade. Signed-off-by: Nihal Jain <nihaljain@apache.org> Signed-off-by: Istvan Toth <stoty@apache.org>

… (addendum) (#6349) Add default implementations of the new methods so that a custom implementation of AsyncBufferedMutatorBuilder will not fail to compile after upgrade. Signed-off-by: Nihal Jain <nihaljain@apache.org> Signed-off-by: Istvan Toth <stoty@apache.org>

Evie Boland added 2 commits July 11, 2024 14:10

Add request attribute support to BufferedMutator

3c259b8

Merge branch 'master' of github.com:apache/hbase into HBASE-28001

b897921

This comment has been minimized.

Sign in to view

Evie Boland added 3 commits July 12, 2024 14:59

Merge branch 'master' of github.com:apache/hbase into HBASE-28001

e5b5963

Fix build issues

23e657e

Fix infinite recursion bug in test

e1e241e

This comment has been minimized.

Sign in to view

Spotless:apply

46aa8b5

This comment has been minimized.

Sign in to view

eab148 marked this pull request as ready for review July 15, 2024 18:41

ndimiduk approved these changes Jul 17, 2024

View reviewed changes

Evie Boland added 4 commits July 17, 2024 14:52

Merge branch 'master' of github.com:apache/hbase into HBASE-28001

a4dbc10

PR comments

3e8c751

Add example to hbase-example

3cecd5a

Remove mutable static fields from test

7abdc55

This comment has been minimized.

Sign in to view

Fix tests

014c1a6

This comment has been minimized.

Sign in to view

Evie Boland added 2 commits August 20, 2024 09:53

Remove unnecessary code for testing

b382a76

Merge branch 'master' of github.com:apache/hbase into HBASE-28001

daf6b79

This comment has been minimized.

Sign in to view

Spotless:apply

1812fd6

This comment has been minimized.

Sign in to view

Remove duplicate dependency

0a04b60

This comment has been minimized.

Sign in to view

ndimiduk merged commit b19ee00 into apache:master Sep 5, 2024
1 check passed

ndimiduk deleted the HBASE-28001 branch September 5, 2024 13:41

ndimiduk pushed a commit to ndimiduk/hbase that referenced this pull request Sep 5, 2024

HBASE-28001: Add request attribute support to BufferedMutator (apache…

59a11d6

…#6076) Co-authored-by: Evie Boland <eboland@hubspot.com> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>

ndimiduk pushed a commit that referenced this pull request Sep 9, 2024

HBASE-28001: Add request attribute support to BufferedMutator (#6076)

c17fef3

Co-authored-by: Evie Boland <eboland@hubspot.com> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>

This was referenced Sep 13, 2024

HBase-28001: Backport #6245

Closed

HBASE-28001: Backport to branch-2 #6246

Merged

ndimiduk pushed a commit that referenced this pull request Sep 17, 2024

HBASE-28001: Add request attribute support to BufferedMutator (#6076)

192f640

Co-authored-by: Evie Boland <eboland@hubspot.com> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>

ndimiduk pushed a commit to ndimiduk/hbase that referenced this pull request Sep 17, 2024

HBASE-28001: Add request attribute support to BufferedMutator (apache…

c8044d6

…#6076) Co-authored-by: Evie Boland <eboland@hubspot.com> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>

ndimiduk pushed a commit that referenced this pull request Sep 17, 2024

HBASE-28001: Add request attribute support to BufferedMutator (#6076)

7d019d9

Co-authored-by: Evie Boland <eboland@hubspot.com> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>

eab148 added a commit to HubSpot/hbase that referenced this pull request Oct 3, 2024

HubSpot Backport HBASE-28001: Add request attribute support to Buffer…

d34b85b

…edMutator (apache#6076) Co-authored-by: Evie Boland <eboland@hubspot.com> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>

eab148 added a commit to HubSpot/hbase that referenced this pull request Oct 3, 2024

HubSpot Backport: HBASE-28001: Add request attribute support to Buffe…

59554ce

…redMutator (apache#6076) Co-authored-by: Evie Boland <eboland@hubspot.com> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>

ndimiduk added a commit to ndimiduk/hbase that referenced this pull request Oct 4, 2024

Revert "HBASE-28001: Add request attribute support to BufferedMutator (…

473ffc2

…apache#6076)" This reverts commit 7d019d9.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HBASE-28001: Add request attribute support to BufferedMutator #6076

HBASE-28001: Add request attribute support to BufferedMutator #6076

eab148 commented Jul 12, 2024 •

edited

Loading

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

ndimiduk left a comment

ndimiduk Jul 17, 2024

ndimiduk Jul 17, 2024

eab148 Jul 17, 2024 •

edited

Loading

ndimiduk commented Jul 17, 2024

eab148 commented Jul 17, 2024

eab148 commented Jul 17, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

eab148 commented Aug 23, 2024

Apache-HBase commented Sep 5, 2024

Apache-HBase commented Sep 5, 2024

ndimiduk commented Sep 5, 2024

HBASE-28001: Add request attribute support to BufferedMutator #6076

HBASE-28001: Add request attribute support to BufferedMutator #6076

Conversation

eab148 commented Jul 12, 2024 • edited Loading

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

ndimiduk left a comment

Choose a reason for hiding this comment

ndimiduk Jul 17, 2024

Choose a reason for hiding this comment

ndimiduk Jul 17, 2024

Choose a reason for hiding this comment

eab148 Jul 17, 2024 • edited Loading

Choose a reason for hiding this comment

ndimiduk commented Jul 17, 2024

eab148 commented Jul 17, 2024

eab148 commented Jul 17, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

eab148 commented Aug 23, 2024

Apache-HBase commented Sep 5, 2024

Apache-HBase commented Sep 5, 2024

ndimiduk commented Sep 5, 2024

eab148 commented Jul 12, 2024 •

edited

Loading

eab148 Jul 17, 2024 •

edited

Loading