HADOOP-16134 001- initial design of a WriteOperationsContext #515
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This adds * a context which is passed round with writes * a parent delete policy as part of this (unused) This PoC shows that adding a new context everywhere is overcomplex as you now need to retrofit it through the stack, even though a (single, shared) WriteOperationsHelper is already passed in Better strategy: * include the WriteOperationsContext in the WriteOperationsHelper; instantiating a new one each time * add a default/configurable delete policy to the FS, *but allow operations to explicitly overwrite this*. Example: completing all the committed work in a job commit, because we can rely on the write of the _SUCCESS file to do the work (so only do it for one file, not every file created) Change-Id: I0de1d4b97fdf4c4f0ece1a27245ba9bb38a29559
| @@ -120,46 +119,43 @@ | |||
| */ | |||
| private final PutTracker putTracker; | |||
|
|
|||
| private final S3AWriteOpContext writeContext; | |||
|
|
|||
| instrumentation.newOutputStreamStatistics(statistics), | ||
| getWriteOperationHelper()); | ||
| } | ||
|
|
| } | ||
|
|
||
| final String key = pathToKey(dst); |
| @@ -3747,4 +3783,5 @@ private void requireSelectSupport(final Path source) throws | |||
| return result; | |||
| } | |||
|
|
|||
|
|
|||
| } | |||
|
💔 -1 overall
This message was automatically generated. |
|
closing this for now; doing a different refactoring |
shanthoosh
pushed a commit
to shanthoosh/hadoop
that referenced
this pull request
Oct 15, 2019
**Changes:**
* Enable all existing standalone integration tests except `TestZkStreamProcessorSession`(`TestZkStreamProcessorSession` is flaky. It spawns `x` StreamProcessors and kills one StreamProcessor through zookeeper session expiration. Sleeps for 5 seconds and proceeds to do validation. If the rebalancing phase takes longer the sleep time, validation fails).
* Remove zookeeper unavailable unit test from LocalApplicationRunner(Race condition in zookeeper shutdown fails other tests). The deleted test will be added back in a separate test class.
* Increase zookeeper server minimum session timeout from 6 seconds to 120 seconds.
* Add assertions to validate if kafka topics setup were successful before the unit tests.
**Validation:**
Verified by running the following script on top of this patch in master branch.
```bash
i=0
while [ $i -lt 50 ]; do
i=`expr $i + 1`
echo "Run " +$i
./gradlew clean :samza-test:test -Dtest.single="TestZkLocalApplicationRunner" --debug --stacktrace >> ~/test-logs_10
./gradlew clean :samza-test:test -Dtest.single="TestZkStreamProcessor" --debug --stacktrace >> ~/test-logs_10
./gradlew clean :samza-test:test -Dtest.single="TestStreamProcessor" --debug --stacktrace >> ~/test-logs_10
./gradlew clean :samza-test:test -Dtest.single="TestZkStreamProcessorFailures" --debug --stacktrace >> ~/test-logs_10
done;
```
**Result:**
```bash
[svenkatasvenkata-ld2 samza]$ grep 'BUILD SUCCESS' ~/test-logs_10 | wc -l
200
[svenkatasvenkata-ld2 samza]$ grep 'BUILD FAIL' ~/test-logs_10 | wc -l
0
```
Author: Shanthoosh Venkataraman <santhoshvenkat1988@gmail.com>
Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>
Reviewers: Xinyu Liu <xinyu@apache.org>
Closes apache#515 from shanthoosh/turn_all_integration_tests_on
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Does not compile
This adds
This PoC shows that adding a new context everywhere is overcomplex as you now need to retrofit it through the stack, even though a (single, shared) WriteOperationsHelper is already passed in
This doesn't compile: I put it together while half-listening to an online talk, and now I've done I've learned enough to say "not the right approach"
Better strategy:
We're also a bit constrained by how the MPU API of HADOOP-13186 tries to be independent of the FS instance -this is one of those cases where it complicates life even more. The FS/FC MUST be the factory for MPU instances.
Change-Id: I0de1d4b97fdf4c4f0ece1a27245ba9bb38a29559