HBASE-24271 Set values in conf/hbase-site.xml that enable running on LocalFileSystem out of the box
#1597
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ndimiduk
requested review from
busbey,
joshelser,
bharathv,
petersomogyi,
infraio,
Apache9 and
apurtell
|
I tried to include everyone who brought opinions to the previous PRs, Jiras, and dev thread. |
ndimiduk
force-pushed
the
24271-update-hbase-site.xml-for-LocalFileSystem
branch
from
9797287
to
5f7bd77
Compare
joshelser
reviewed
Apr 27, 2020
There was a problem hiding this comment.
I'm surprised to not see the switch to change file:// to refer to RawLocalFileSystem instead of LocalFileSystem here. I assume that means that the system generally works OK? My memory is that doing WAL replay will cause trouble w/o RawLocalFS.
If you're just laying groundwork, that's also fine.
hbase-common/src/main/java/org/apache/hadoop/hbase/util/CommonFSUtils.java
Outdated
Show resolved
Hide resolved
saintstack
approved these changes
Apr 27, 2020
hbase-common/src/main/java/org/apache/hadoop/hbase/util/CommonFSUtils.java
Outdated
Show resolved
Hide resolved
| final boolean value = false; | ||
| LOG.warn("Cannot enforce durability guarantees while running on {}. Setting {}={} for" | ||
| + " this FileSystem.", fs.getUri(), UNSAFE_STREAM_CAPABILITY_ENFORCE, value); | ||
| fs.getConf().setBoolean(UNSAFE_STREAM_CAPABILITY_ENFORCE, value); |
| @@ -426,7 +386,7 @@ You can stop HBase the same way as in the <<quickstart,quickstart>> procedure, u | |||
|
|
|||
|
|
|||
| [[quickstart_fully_distributed]] | |||
| === Advanced - Fully Distributed | |||
| === Fully Distributed for Production | |||
|
🎊 +1 overall
This message was automatically generated. |
Nope, not touching any of that. I have no intention of making a |
|
🎊 +1 overall
This message was automatically generated. |
Apache9
approved these changes
Apr 28, 2020
hbase-common/src/main/java/org/apache/hadoop/hbase/util/CommonFSUtils.java
Outdated
Show resolved
Hide resolved
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
ndimiduk
commented
Apr 29, 2020
conf/hbase-site.xml
Outdated
| --> | ||
| <property> | ||
| <name>hbase.tmp.dir</name> | ||
| <value>./tmp</value> |
There was a problem hiding this comment.
Hadoop Configuration supports environment variables. Let me make this even my explicit by changing this value to ${env.HBASE_HOME:-.}/tmp.
ndimiduk
force-pushed
the
24271-update-hbase-site.xml-for-LocalFileSystem
branch
from
5f7bd77
to
69a9386
Compare
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
@joshelser @busbey other thoughts here? |
joshelser
approved these changes
Apr 30, 2020
There was a problem hiding this comment.
You're doing good work :)
I think the natural progression is to keep pulling on this thread. With LocalFileSystem, things will assuredly be finicky around stop/start of RegionServers (especially by our tests) with the lack of that hflush. I understand if that's not your goal now, but it sounds like it could be a big win to try to flip some tests over and run with less overhead (from minidfscluster). If not you, someone else we can convince, maybe :)
busbey
reviewed
May 1, 2020
| @@ -426,7 +386,7 @@ You can stop HBase the same way as in the <<quickstart,quickstart>> procedure, u | |||
|
|
|||
|
|
|||
| [[quickstart_fully_distributed]] | |||
| === Advanced - Fully Distributed | |||
| === Fully Distributed for Production | |||
|
|
|||
| In reality, you need a fully-distributed configuration to fully test HBase and to use it in real-world scenarios. | |||
| In a distributed configuration, the cluster contains multiple nodes, each of which runs one or more HBase daemon. | |||
There was a problem hiding this comment.
We should make clear in this paragraph that this quickstart covers a necessary but not sufficient topic for a system that is "production ready."
Just to be clear that we are only explaining how to get a distributed HBase on top of a distributed filesystem and there exist other topics that need to be considered for a production deployment. Folks should still read through the section "The Important Configurations" and should ensure they have monitoring of metrics and log aggregation, for example.
There was a problem hiding this comment.
Maybe drop my addition of "for Production" then? I thought it odd that our default operating configuration was flagged as "advanced". I agree there's more to a production deploy than just these configs. However, I also don't think it's our job to be prescriptive of what a production deployment looks like, re: monitoring, metrics, log aggregation, &c. I think anything more than a paragraph or two of "recommended" or "strongly encouraged" supporting infrastructure is beyond the scope of our document.
Any further input re: what you would like to see for a +1 @busbey ? It sounds to me from your comment that there's a last mile that I'm missing. I'd rather put this to bed than need revisit it. |
ndimiduk
force-pushed
the
24271-update-hbase-site.xml-for-LocalFileSystem
branch
from
69a9386
to
f825b91
Compare
…n `LocalFileSystem` out of the box Simplify the new user experience shipping a configuration that enables a fresh checkout or tarball distribution to run in standalone mode without direct user configuration. This change restores the behavior we had when running on Hadoop 2.8 and earlier. Patch for master includes an update to the book. This change will be omitted when backporting to earlier branches. Signed-off-by: stack <stack@apache.org> Signed-off-by: Josh Elser <elserj@apache.org> Signed-off-by: Duo Zhang <zhangduo@apache.org>
ndimiduk
force-pushed
the
24271-update-hbase-site.xml-for-LocalFileSystem
branch
from
f825b91
to
9e8107b
Compare
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Simplify the new user experience shipping a configuration that enables
a fresh checkout or tarball distribution to run in standalone mode
without direct user configuration. This change restores the behavior
we had when running on Hadoop 2.8 and earlier.
Patch for master includes an update to the book. This change will be
omitted when backporting to earlier branches.