HDDS-14343. Apache Knox integration documentation and Docker Compose demo by jojochuang · Pull Request #490 · apache/ozone-site

jojochuang · 2026-07-14T03:48:43Z

Summary

Add administrator guide for configuring Apache Knox with Ozone (OM, SCM, Recon UIs and HttpFS), including user access instructions and a Docker Compose quickstart under static/compose/knox-ozone/.
Fix Knox browser login in Docker by mounting knoxsso.xml with LDAP pointed at the knox-ldap service (default image topology uses localhost).
Enable the Knox integration card on the homepage and cross-link from the HttpFS user guide.

Jira

https://issues.apache.org/jira/browse/HDDS-14343

Test plan

pnpm build passes with no broken links
pnpm exec yamllint static/compose/knox-ozone/docker-compose.yaml passes
docker compose up -d --scale datanode=1 in static/compose/knox-ozone/ starts all services
Knox homepage login works with guest / guest-password
OM, SCM, and Recon UIs reachable via Knox (OM/SCM need ?host=http://...)
HttpFS LISTSTATUS works through Knox at /gateway/ozone/httpfs/v1/

…demo. Document Knox configuration for Ozone UIs and HttpFS, add a runnable compose stack with browser-login fix, and link the integration from the homepage and HttpFS guide. Co-authored-by: Cursor <cursoragent@cursor.com>

jojochuang · 2026-07-14T05:32:03Z

I had Cursor follow the documented steps and verified correctness.

chihsuan

Thanks for adding this guide! @jojochuang

I tested the full quickstart locally and confirmed that the Compose stack, Knox authentication, Ozone UIs, and HttpFS endpoint work as documented.

I left a few inline comments, mostly around version requirements and HttpFS identity handling. Overall, the integration looks great.

chihsuan · 2026-07-16T10:27:19Z


-**TODO:** File a subtask under [HDDS-9859](https://issues.apache.org/jira/browse/HDDS-9859) and complete this page or section.
-**TODO:** Uncomment link to this page in src/pages/index.js
+[Apache Knox](https://knox.apache.org/) is a reverse proxy gateway for Hadoop ecosystem REST APIs and web UIs. Knox 2.0.0 and later can proxy Ozone services, giving users a single HTTPS entry point while hiding internal cluster hostnames and ports.


OZONE-HTTPFS is not available in Knox 2.0.0. It was added by KNOX-2914 (apache/knox#765) after the 2.0.0 release, and the service definition first appears in the v2.1.0 tag.

Could we require Knox 2.1.0 here, or clarify that 2.0.0 supports the Ozone UIs while HttpFS proxying requires 2.1.0?

Fixed in eb709d3. Split the intro and prerequisites: Knox 2.0.0+ for Ozone UIs (OM/SCM/Recon), Knox 2.1.0+ when proxying HttpFS (KNOX-2914).

chihsuan · 2026-07-16T10:29:44Z

+      <<: *common-config
+      CORE-SITE.XML_fs.defaultFS: ofs://om
+      CORE-SITE.XML_httpfs.proxyuser.knoxuser.hosts: "*"
+      CORE-SITE.XML_httpfs.proxyuser.knoxuser.groups: "*"


These variables are converted into properties in core-site.xml, but Ozone HttpFS loads its server settings from httpfs-default.xml and httpfs-site.xml.

https://github.com/apache/ozone/blob/1a9439452d9aae2ffd9ed4d20e8ac1f3a9f98864/hadoop-ozone/httpfsgateway/src/main/java/org/apache/ozone/lib/server/Server.java#L49-L52

HttpFSAuthenticationFilter#getProxyuserConfiguration() reads httpfs.proxyuser.* from that HttpFS configuration, so these properties will not take effect here. Should these use the HTTPFS-SITE.XML_ prefix instead, or should the compose stack mount an httpfs-site.xml?

Fixed in eb709d3. Added conf/httpfs-site.xml with the proxy-user settings and mount it on the HttpFS service at /opt/hadoop/etc/hadoop/httpfs-site.xml. Removed the incorrect CORE-SITE.XML_httpfs.proxyuser.* env vars and updated the doc accordingly.

chihsuan · 2026-07-16T10:33:52Z

+
+# Optional image that bakes the Ozone topology into the official Knox gateway
+# image. The default docker-compose.yaml mounts the topology instead.
+FROM apache/knox:latest


nit: should we consider pinning to a verified version (e.g. apache/knox:2.1.0) instead of latest, so the documented steps don't silently break on a future Knox major release?

Same for the KNOX_IMAGE_VERSION:-latest default in docker-compose.yaml.

Fixed in eb709d3. Default KNOX_IMAGE_VERSION is now 2.1.0 in compose (still overridable via env var). The optional Dockerfile pins apache/knox:2.1.0 as well.

chihsuan · 2026-07-16T10:35:22Z

+
+```bash
+curl -ik -u guest:guest-password \
+  "https://localhost:8443/gateway/ozone/httpfs/v1/?op=LISTSTATUS&user.name=hadoop"


nit: This section (general usage, above the Docker quickstart) uses https://localhost:8443 while the surrounding examples use <knox-host>. Would it make sense to use <knox-host> here too?

Fixed in eb709d3. Updated the general-usage curl example to use https://<knox-host>:8443 for consistency.

chihsuan · 2026-07-16T10:39:57Z

+  "https://localhost:8443/gateway/ozone/httpfs/v1/?op=LISTSTATUS&user.name=hadoop"
+```
+
+The `user.name` parameter specifies the Ozone user to impersonate. Knox authenticates the caller; HttpFS performs the operation as the proxied user.


I tested the compose stack locally, and the HttpFS access log shows Knox rewrites user.name to the authenticated Knox user regardless of what the client sends:

GET /webhdfs/v1/?op=LISTSTATUS&user.name=guest

This is the request HttpFS received, even though the client sent user.name=hadoop. So "The user.name parameter specifies the Ozone user to impersonate" isn't accurate in this setup.

Suggested change

The `user.name` parameter specifies the Ozone user to impersonate. Knox authenticates the caller; HttpFS performs the operation as the proxied user.

The `user.name` parameter specifies the Ozone user to impersonate. Knox authenticates access to the gateway, while this quickstart’s HttpFS backend uses the `user.name` query parameter through pseudo authentication.

Fixed in eb709d3. Replaced the impersonation paragraph with your suggested wording about Knox rewriting user.name to the authenticated Knox user.

chihsuan · 2026-07-16T10:41:46Z

+  OZONE-SITE.XML_hdds.scm.safemode.healthy.pipeline.pct: "0"
+  no_proxy: om,recon,scm,s3g,httpfs,knox-ldap,knox-gateway,localhost,127.0.0.1
+
+version: "3"


nit: Docker Compose v2 treats the top-level version field as obsolete and ignores it with a warning. We can probably remove this.

https://docs.docker.com/reference/compose-file/version-and-name/

Fixed in eb709d3. Removed the top-level version: "3" field.

Split Knox 2.0.0 (UIs) vs 2.1.0 (HttpFS) requirements, mount httpfs-site.xml for proxy-user settings, default Knox image to 2.1.0, and fix HttpFS impersonation wording and curl example consistency. Co-authored-by: Cursor <cursoragent@cursor.com>

Match the license comment format used by the other compose XML files so the static license CI check passes. Co-authored-by: Cursor <cursoragent@cursor.com>

jojochuang · 2026-07-17T22:00:06Z

All CI checks are green on the latest commit (899a183). Ready for merge from my side — addressed chihsuan's review feedback in eb709d3 and fixed the license header in 899a183.

jojochuang · 2026-07-17T23:02:04Z

Merged. Thanks for review, @chihsuan

jojochuang changed the title ~~HDDS-15850. Publish Ozone CLI Agent Skill on ozone.apache.org~~ HDDS-15850 and HDDS-14343. Ozone CLI skill and Knox integration docs Jul 14, 2026

jojochuang changed the title ~~HDDS-15850 and HDDS-14343. Ozone CLI skill and Knox integration docs~~ HDDS-15850. Publish Ozone CLI skill and Knox integration docs Jul 14, 2026

jojochuang force-pushed the HDDS-15850 branch from ff1ab72 to 56b464f Compare July 14, 2026 05:25

jojochuang changed the title ~~HDDS-15850. Publish Ozone CLI skill and Knox integration docs~~ HDDS-14343. Apache Knox integration documentation and Docker Compose demo Jul 14, 2026

jojochuang requested a review from dombizita July 14, 2026 05:29

jojochuang marked this pull request as ready for review July 14, 2026 05:29

chihsuan reviewed Jul 16, 2026

View reviewed changes

jojochuang and others added 2 commits July 17, 2026 14:50

Fix ASF license header in knox-ozone httpfs-site.xml.

899a183

Match the license comment format used by the other compose XML files so the static license CI check passes. Co-authored-by: Cursor <cursoragent@cursor.com>

jojochuang merged commit df37395 into apache:master Jul 17, 2026
10 checks passed

	The `user.name` parameter specifies the Ozone user to impersonate. Knox authenticates the caller; HttpFS performs the operation as the proxied user.
	The `user.name` parameter specifies the Ozone user to impersonate. Knox authenticates access to the gateway, while this quickstart’s HttpFS backend uses the `user.name` query parameter through pseudo authentication.

Uh oh!

Conversation

jojochuang commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Jira

Test plan

Uh oh!

jojochuang commented Jul 14, 2026

Uh oh!

chihsuan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jojochuang commented Jul 17, 2026

Uh oh!

Uh oh!

jojochuang commented Jul 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jojochuang commented Jul 14, 2026 •

edited

Loading