Skip to content

Add topology provider jar to Hadoop image. #509

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
soenkeliebau merged 3 commits into from
Nov 23, 2023
Merged

Add topology provider jar to Hadoop image. #509

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c2afcde
Add topology provider jar to Hadoop image.
soenkeliebau Nov 22, 2023
e42f17c
Added changelog entry
soenkeliebau Nov 23, 2023
c0985ce
Changed from using ADD to using curl --fail to ensure properly aborting.
soenkeliebau Nov 23, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ All notable changes to this project will be documented in this file.

### Added

- hadoop: Added Stackable topology provider jar to enable k8s-based rack awareness ([#509])
- hadoop: Add all necessary components to the image to mount HDFS using FUSE ([#400])
- hbase: Add hbase-operator-tools ([#497], [#498]).
- java-base: Add needed tzdata-java package ([#425]).
Expand Down Expand Up @@ -109,6 +110,7 @@ All notable changes to this project will be documented in this file.
[#498]: https://github.com/stackabletech/docker-images/pull/498
[#499]: https://github.com/stackabletech/docker-images/pull/499
[#505]: https://github.com/stackabletech/docker-images/pull/505
[#509]: https://github.com/stackabletech/docker-images/pull/509

## [23.7.0] - 2023-07-14

Expand Down
12 changes: 8 additions & 4 deletions conf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,25 +62,29 @@
"product": "3.2.2",
"java-base": "11",
"java": "11",
"jmx_exporter": "0.20.0"
"jmx_exporter": "0.20.0",
"topology_provider": "0.1.0"
},
{
"product": "3.2.4",
"java-base": "11",
"java": "11",
"jmx_exporter": "0.20.0"
"jmx_exporter": "0.20.0",
"topology_provider": "0.1.0"
},
{
"product": "3.3.4",
"java-base": "11",
"java": "11",
"jmx_exporter": "0.20.0"
"jmx_exporter": "0.20.0",
"topology_provider": "0.1.0"
},
{
"product": "3.3.6",
"java-base": "11",
"java": "11",
"jmx_exporter": "0.20.0"
"jmx_exporter": "0.20.0",
"topology_provider": "0.1.0"
},
],
},
Expand Down
7 changes: 7 additions & 0 deletions hadoop/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ FROM stackable/image/java-base AS builder
ARG PRODUCT
ARG JAVA
ARG JMX_EXPORTER
ARG TOPOLOGY_PROVIDER

# https://github.com/hadolint/hadolint/wiki/DL4006
SHELL ["/bin/bash", "-o", "pipefail", "-c"]
Expand Down Expand Up @@ -139,6 +140,12 @@ WORKDIR /stackable

COPY --chown=stackable:stackable --from=builder /stackable/hadoop-${PRODUCT} /stackable/hadoop-${PRODUCT}/
COPY --chown=stackable:stackable --from=builder /stackable/jmx /stackable/jmx/

# The topology provider provides rack awareness functionality for HDFS by allowing users to specify Kubernetes
# labels to build a rackID from
# source code is at: https://github.com/stackabletech/hdfs-topology-provider
RUN curl --fail -L -s -S https://repo.stackable.tech/repository/packages/hdfs-topology-provider/topology-provider-${TOPOLOGY_PROVIDER}.jar -o /stackable/hadoop/share/hadoop/tools/lib/ --chown=stackable:stackable

RUN ln -s /stackable/hadoop-${PRODUCT} /stackable/hadoop
COPY hadoop/stackable/fuse_dfs_wrapper /stackable/hadoop/bin

Expand Down