[ozone] Dedicated Ozone browser in Hue #3189

Harshg999 · 2023-02-07T20:42:51Z

What changes were proposed in this pull request?

Adding dedicated Ozone browser in Hue like other file-systems.

The PR can be divided into following sub-sections:

Adding ozone related configuration and defining the directory structure.
- Almost similar to the existing HDFS config.
Add the OFS client extending the already existing WebHDFS client.
Default to Ozone root for all users.
- As of now, Ozone is not having the concept of user home directories. If they have it the future, we can make the necessary changes.
Give necessary permissions according to the Hue user and groups.
Checking all the operations and updating for the ozone specific ones.
Having a dedicated upload handler using the /create API because Ozone does not support /append yet.
- This restricts us to upload the file content as one chunk only.
- Uploads via the /create API is slow generally, therefore max size (assuming chunk size allows it) also depends on network speed and Hue connection timeout.
Add the supporting UI changes.
- Icons and detailed branding will be done in other PRs.
Make Importer work with Ozone path.
- Hive and Impala needs fs_defaultfs in the path for queries like LOAD DATA INPATH "ofs://....
- If Ozone is in HA mode, then fs_defaultfs will have a service ID else it will have the URI of the Ozone Manager.
Add new unit tests and update the existing ones for the Ozone FS.

How was this patch tested?

Via dedicated and existing unit tests.
Tested manually by connecting to the un-secure Ozone service supporting HttpfS APIs.
Tested E2E in a secure cluster having Ozone service supporting HttpfS APIs.
- Existing Hue in that cluster was swapped with the latest Hue containing these changes. Hue was working fine in the secure cluster OOTB.

Harshg999 · 2023-03-08T12:15:37Z

The PR is ready for review.
@JohanAhlen @ranade1

Harshg999 · 2023-03-08T13:29:17Z

The PR description has been added.
The changes of the PR has been tested both via unit tests and E2E tests.

JohanAhlen

Super nice!

This one is touching quite some legacy UI code and some things can be shortened/improved but I decided not to comment about that as we can better spend time cleaning up as part of moving the File Browser to React.

A few nits but all-in-all looks good to me!

JohanAhlen · 2023-03-08T13:47:59Z

apps/filebrowser/src/filebrowser/templates/listdir_components.mako

+      });
+
+      self.isOFSVol = ko.pureComputed(function () {
+        return self.isOFS() && self.currentPath().split("/").length === 3 && self.currentPath().split("/")[2] !== '';


Nit: Instead of split, why not .. && self.currentPath().toLowerCase() === 'ofs://'?

I didn't get this.
Here we are checking, that current path is at volume level (ofs://volume) or not.
so I was splitting the path like this -> ['ofs:', '', 'volume']

desktop/core/src/desktop/js/ko/bindings/ace/ko.aceEditor.js

amitsrivastava

Awesome work folks.

I have put up some questions. Please check.

desktop/conf.dist/hue.ini

desktop/core/src/desktop/lib/fs/ozone/__init__.py

desktop/core/src/desktop/lib/fs/ozone/ofs.py

desktop/core/src/desktop/lib/fs/ozone/upload.py

apps/filebrowser/src/filebrowser/templates/listdir_components.mako

desktop/libs/indexer/src/indexer/indexers/sql.py

desktop/core/src/desktop/lib/fs/proxyfs.py

- Add ofs as a separate fs in proxyfs - Add ozone to user and group permissions - Update APIs with ofs - Default users to ofs root directory

- Check for fs_defaultFS null scenarios and raise error. - Use urlparse for better code readability. - Improve comments.

Harshg999 · 2023-03-10T05:45:45Z

PR is rebased, merging now.

(cherry picked from commit 6b03c71) (cherry picked from commit b5c2255) Change-Id: I24b7be3aefba808abe949dea9e486f05dbb0727d (cherry picked from commit 0cc32ca)

(cherry picked from commit 6cc167a) (cherry picked from commit cca6b8e) Change-Id: I8d6dcce200f0f8b217da7e682f691ba66aa918da (cherry picked from commit 04a6290)

(cherry picked from commit 3a067c4) (cherry picked from commit 045903b) Change-Id: I49de9d2e91c83fd0a138d544062469a687d1546b (cherry picked from commit 36a356c)

(cherry picked from commit 05e4855) (cherry picked from commit ac4d32d) Change-Id: I284ae517df9d217bb2c200ea1dfaa78a86e77d0d (cherry picked from commit 8386b8f)

(cherry picked from commit de3a3e6) (cherry picked from commit ce14518) Change-Id: I7d3353bf1648afc403f8e2ed39cd450077422617 (cherry picked from commit 0029f3d)

(cherry picked from commit 957fe50) (cherry picked from commit 71a90da) Change-Id: Ibce61134a648b6b1eb5c484fa7aaf1b117b28b0b (cherry picked from commit df1cf50)

- Add ofs as a separate fs in proxyfs - Add ozone to user and group permissions - Update APIs with ofs - Default users to ofs root directory (cherry picked from commit f050809) (cherry picked from commit 2e78a51) Change-Id: I1f1e7d03f9bb603927f3198d4b981ce062cae1f1 (cherry picked from commit 6034816)

…3189) (cherry picked from commit 08d1474) (cherry picked from commit ea09281) Change-Id: I562f40a16948dd9b2fa997c30ee3aaed0133c8eb (cherry picked from commit fd42b1c)

- Check for fs_defaultFS null scenarios and raise error. - Use urlparse for better code readability. - Improve comments. (cherry picked from commit 731cc5c) (cherry picked from commit e11b8b0) Change-Id: If0058097856d33937b74f4741dfa08654cfc02fa (cherry picked from commit 2c5a148)

(cherry picked from commit 6b03c71) (cherry picked from commit b5c2255) Change-Id: I24b7be3aefba808abe949dea9e486f05dbb0727d

(cherry picked from commit 6cc167a) (cherry picked from commit cca6b8e) Change-Id: I8d6dcce200f0f8b217da7e682f691ba66aa918da

(cherry picked from commit 3a067c4) (cherry picked from commit 045903b) Change-Id: I49de9d2e91c83fd0a138d544062469a687d1546b

(cherry picked from commit 05e4855) (cherry picked from commit ac4d32d) Change-Id: I284ae517df9d217bb2c200ea1dfaa78a86e77d0d

(cherry picked from commit de3a3e6) (cherry picked from commit ce14518) Change-Id: I7d3353bf1648afc403f8e2ed39cd450077422617

(cherry picked from commit 957fe50) (cherry picked from commit 71a90da) Change-Id: Ibce61134a648b6b1eb5c484fa7aaf1b117b28b0b

- Add ofs as a separate fs in proxyfs - Add ozone to user and group permissions - Update APIs with ofs - Default users to ofs root directory (cherry picked from commit f050809) (cherry picked from commit 2e78a51) Change-Id: I1f1e7d03f9bb603927f3198d4b981ce062cae1f1

…3189) (cherry picked from commit 08d1474) (cherry picked from commit ea09281) Change-Id: I562f40a16948dd9b2fa997c30ee3aaed0133c8eb

- Check for fs_defaultFS null scenarios and raise error. - Use urlparse for better code readability. - Improve comments. (cherry picked from commit 731cc5c) (cherry picked from commit e11b8b0) Change-Id: If0058097856d33937b74f4741dfa08654cfc02fa

Harshg999 self-assigned this Feb 7, 2023

Harshg999 marked this pull request as draft February 7, 2023 21:06

Harshg999 changed the title ~~[ozone] Add ozone config and improve dir structure~~ [ozone] Add ozone config and improve directory structure Feb 7, 2023

agl29 force-pushed the ozone-conf branch 2 times, most recently from 3ad1636 to 12f1468 Compare February 22, 2023 11:31

Harshg999 force-pushed the ozone-conf branch from 12f1468 to 5dd10ef Compare February 23, 2023 13:41

Harshg999 changed the title ~~[ozone] Add ozone config and improve directory structure~~ [ozone] Ozone V1 skeleton Mar 1, 2023

agl29 force-pushed the ozone-conf branch from b7ac00c to 55061d2 Compare March 3, 2023 04:57

bjornalm requested review from bjornalm and removed request for bjornalm March 3, 2023 09:37

agl29 force-pushed the ozone-conf branch from eb975d3 to b2963bf Compare March 6, 2023 11:33

Harshg999 marked this pull request as ready for review March 7, 2023 06:58

Harshg999 requested review from JohanAhlen, bjornalm, wing2fly, agl29, amitsrivastava, athithyaaselvam and ranade1 March 7, 2023 07:22

agl29 force-pushed the ozone-conf branch from be6297d to 898bc8a Compare March 7, 2023 15:26

Harshg999 force-pushed the ozone-conf branch 2 times, most recently from 65e10e0 to 4588703 Compare March 8, 2023 12:11

Harshg999 changed the title ~~[ozone] Ozone V1 skeleton~~ [ozone] Dedicated Ozone browser in Hue Mar 8, 2023

JohanAhlen approved these changes Mar 8, 2023

View reviewed changes

amitsrivastava requested changes Mar 8, 2023

View reviewed changes

Harshg999 force-pushed the ozone-conf branch from 9393ae7 to 8893364 Compare March 9, 2023 10:56

amitsrivastava approved these changes Mar 10, 2023

View reviewed changes

[ozone] Add initial config under desktop

7bb5606

agl29 and others added 6 commits March 10, 2023 11:14

[ozone] Add ofs upload filehandler

d39182c

[ozone] Add UI changes

f86d33a

[ozone] Add fs_defaultfs in path in importer

fd2a50f

[ozone] Connect ofs like other filesystems

73c6625

- Add ofs as a separate fs in proxyfs - Add ozone to user and group permissions - Update APIs with ofs - Default users to ofs root directory

[ozone] Add unit tests and update existing tests for ofs

763040a

[ozone] Improve fs_defaultfs check and uploadhandler

5657e4e

- Check for fs_defaultFS null scenarios and raise error. - Use urlparse for better code readability. - Improve comments.

Harshg999 force-pushed the ozone-conf branch from 24b8ab5 to 5657e4e Compare March 10, 2023 05:45

Harshg999 enabled auto-merge (rebase) March 10, 2023 05:46

Harshg999 merged commit 731cc5c into master Mar 10, 2023

Harshg999 deleted the ozone-conf branch March 10, 2023 06:04

agl29 mentioned this pull request Mar 10, 2023

[ozone] Better naming for abspath var and using rename API for rename operation #3225

Merged

agl29 added a commit that referenced this pull request Jul 24, 2023

CDPD-49779: [ozone] add ofs client (#3189)

cf7351f

(cherry picked from commit 6cc167a) (cherry picked from commit cca6b8e) Change-Id: I8d6dcce200f0f8b217da7e682f691ba66aa918da (cherry picked from commit 04a6290)

agl29 added a commit that referenced this pull request Jul 24, 2023

CDPD-49776: [ozone] Add UI changes (#3189)

18b1fc6

(cherry picked from commit de3a3e6) (cherry picked from commit ce14518) Change-Id: I7d3353bf1648afc403f8e2ed39cd450077422617 (cherry picked from commit 0029f3d)

athithyaaselvam pushed a commit that referenced this pull request Mar 14, 2024

CDPD-49313: [ozone] Add initial config under desktop (#3189)

0cc32ca

(cherry picked from commit 6b03c71) (cherry picked from commit b5c2255) Change-Id: I24b7be3aefba808abe949dea9e486f05dbb0727d

athithyaaselvam pushed a commit that referenced this pull request Mar 14, 2024

CDPD-49779: [ozone] add ofs client (#3189)

04a6290

(cherry picked from commit 6cc167a) (cherry picked from commit cca6b8e) Change-Id: I8d6dcce200f0f8b217da7e682f691ba66aa918da

athithyaaselvam pushed a commit that referenced this pull request Mar 14, 2024

CDPD-50020: [Ozone] Add ofsstat separately (#3189)

36a356c

(cherry picked from commit 3a067c4) (cherry picked from commit 045903b) Change-Id: I49de9d2e91c83fd0a138d544062469a687d1546b

athithyaaselvam pushed a commit that referenced this pull request Mar 14, 2024

CDPD-49900: [ozone] Add ofs upload filehandler (#3189)

8386b8f

(cherry picked from commit 05e4855) (cherry picked from commit ac4d32d) Change-Id: I284ae517df9d217bb2c200ea1dfaa78a86e77d0d

athithyaaselvam pushed a commit that referenced this pull request Mar 14, 2024

CDPD-49776: [ozone] Add UI changes (#3189)

0029f3d

(cherry picked from commit de3a3e6) (cherry picked from commit ce14518) Change-Id: I7d3353bf1648afc403f8e2ed39cd450077422617

athithyaaselvam pushed a commit that referenced this pull request Mar 14, 2024

CDPD-50084: [ozone] Add fs_defaultfs in path in importer (#3189)

df1cf50

(cherry picked from commit 957fe50) (cherry picked from commit 71a90da) Change-Id: Ibce61134a648b6b1eb5c484fa7aaf1b117b28b0b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ozone] Dedicated Ozone browser in Hue #3189

[ozone] Dedicated Ozone browser in Hue #3189

Harshg999 commented Feb 7, 2023 •

edited

Harshg999 commented Mar 8, 2023

Harshg999 commented Mar 8, 2023

JohanAhlen left a comment

JohanAhlen Mar 8, 2023

agl29 Mar 9, 2023

amitsrivastava left a comment

Harshg999 commented Mar 10, 2023

[ozone] Dedicated Ozone browser in Hue #3189

[ozone] Dedicated Ozone browser in Hue #3189

Conversation

Harshg999 commented Feb 7, 2023 • edited

What changes were proposed in this pull request?

How was this patch tested?

Harshg999 commented Mar 8, 2023

Harshg999 commented Mar 8, 2023

JohanAhlen left a comment

Choose a reason for hiding this comment

JohanAhlen Mar 8, 2023

Choose a reason for hiding this comment

agl29 Mar 9, 2023

Choose a reason for hiding this comment

amitsrivastava left a comment

Choose a reason for hiding this comment

Harshg999 commented Mar 10, 2023

Harshg999 commented Feb 7, 2023 •

edited