Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HDDS-1627. Make the version of the used hadoop-runner configurable #886

Merged
merged 1 commit into from
Jun 11, 2019

Conversation

elek
Copy link
Member

@elek elek commented Jun 3, 2019

During an offline discussion with [~arp] and [~eyang] we agreed that it could be more safe to fix the tag of the used hadoop-runner images during the releases.

It also requires fix tags from hadoop-runner, but after that it's possible to use the fixed tags.

This patch makes it possible to define the required version/tag in pom.xml

  1. the default hadoop-runner.version is added to all .env files during the build
  2. If a variable is added to the .env, it can be used from docker-compose files AND can be overridden by environment variables (it makes it possible to define custom version during a local run)

See: https://issues.apache.org/jira/browse/HDDS-1627

@elek elek added the ozone label Jun 3, 2019
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 48 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
0 yamllint 0 yamllint was not available.
+1 @author 0 The patch does not contain any @author tags.
-1 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 mvninstall 626 trunk passed
+1 compile 297 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 1789 branch has no errors when building and testing our client artifacts.
+1 javadoc 185 trunk passed
_ Patch Compile Tests _
+1 mvninstall 710 the patch passed
+1 compile 387 the patch passed
+1 javac 387 the patch passed
+1 mvnsite 0 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 xml 3 The patch has no ill-formed XML file.
+1 shadedclient 810 patch has no errors when building and testing our client artifacts.
+1 javadoc 222 the patch passed
_ Other Tests _
-1 unit 312 hadoop-hdds in the patch failed.
-1 unit 2743 hadoop-ozone in the patch failed.
+1 asflicense 67 The patch does not generate ASF License warnings.
7518
Reason Tests
Failed junit tests hadoop.hdds.scm.block.TestBlockManager
hadoop.ozone.TestStorageContainerManager
hadoop.ozone.container.common.statemachine.commandhandler.TestBlockDeletion
hadoop.ozone.client.rpc.TestWatchForCommit
hadoop.ozone.TestContainerStateMachineIdempotency
hadoop.ozone.client.rpc.TestSecureOzoneRpcClient
hadoop.ozone.om.TestScmSafeMode
hadoop.ozone.TestMiniChaosOzoneCluster
hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient
hadoop.ozone.client.rpc.TestBlockOutputStream
hadoop.ozone.client.rpc.TestBCSID
hadoop.ozone.scm.node.TestQueryNode
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-886/1/artifact/out/Dockerfile
GITHUB PR #886
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml yamllint
uname Linux 6fc992229da9 4.4.0-143-generic #169~14.04.2-Ubuntu SMP Wed Feb 13 15:00:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 2185249
Default Java 1.8.0_212
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-886/1/artifact/out/patch-unit-hadoop-hdds.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-886/1/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-886/1/testReport/
Max. process+thread count 4448 (vs. ulimit of 5500)
modules C: hadoop-ozone/dist U: hadoop-ozone/dist
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-886/1/console
versions git=2.7.4 maven=3.3.9
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@anuengineer anuengineer merged commit 96e1e41 into apache:trunk Jun 11, 2019
bshashikant pushed a commit to bshashikant/hadoop that referenced this pull request Jul 10, 2019
shanthoosh added a commit to shanthoosh/hadoop that referenced this pull request Oct 15, 2019
…ointTool.

 Currently, each run of a `CheckpointTool` requires the `TaskModel` from the `JobModel` for updating the checkpoints. `JobModel` generation involves reading the `SystemStreamMetadata`, `SystemStreamPartitionMetadata` of the input streams of the job. Post samza 1.0, this would require the entire configuration bag stored in coordinator stream.

As a followup to `SAMZA-2059`, this PR changes the `CheckpointTool` to read the configuration from the coordinator stream and use it when generating the `JobModel`.

Author: Shanthoosh Venkataraman <spvenkat@usc.edu>

Reviewers: Cameron L<calee@linkedin.com>

Closes apache#886 from shanthoosh/read_config_from_coordinator_stream_for_checkpoint_tool and squashes the following commits:

c0a68ec9 [Shanthoosh Venkataraman] Address review comments.
910cfaca [Shanthoosh Venkataraman] Read configuration from coordinator stream in checkpoint tool.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
3 participants