Deploy UI within fidesops #625

TheAndrewJackson · 2022-06-09T22:25:21Z

Purpose

Build and serve the Admin UI within fidesops as a static website.

Changes

Update Dockerfile to build website and copy files into the project
Add startup event to serve the website if it's toggled on
Fix a few miscellaneous issues with serving the frontend from within fidesops
Refactor Head portion of pages into single FidesHead component to remove duplication

Checklist

Ticket

Fixes #528 #529

conceptualshark · 2022-06-10T20:56:48Z

docs/fidesops/docs/guides/configuration_reference.md

+| Admin UI Variables|---|---|---|---|---|
+|`ENABLED` | `FIDESOPS__ADMIN_UI__ENABLED` | bool | False | True | Toggle whether the `/static` file directory is mounted to serve the Admin UI


Thank you for this! 👍

eastandwestwind · 2022-06-13T14:22:00Z

@TheAndrewJackson I'm getting this build err locally when trying to make server, any ideas?

Building fidesops
[+] Building 1.2s (6/6) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                   0.0s
 => => transferring dockerfile: 2.08kB                                                                                                                                                                 0.0s
 => [internal] load .dockerignore                                                                                                                                                                      0.0s
 => => transferring context: 208B                                                                                                                                                                      0.0s
 => ERROR [internal] load metadata for docker.io/library/python:3.9.13-slim-buster                                                                                                                     1.1s
 => ERROR [internal] load metadata for docker.io/library/node:16                                                                                                                                       1.1s
 => [auth] library/python:pull token for registry-1.docker.io                                                                                                                                          0.0s
 => [auth] library/node:pull token for registry-1.docker.io                                                                                                                                            0.0s
------
 > [internal] load metadata for docker.io/library/python:3.9.13-slim-buster:
------
------
 > [internal] load metadata for docker.io/library/node:16:
------
failed to solve with frontend dockerfile.v0: failed to create LLB definition: failed to authorize: rpc error: code = Unknown desc = failed to fetch oauth token: unexpected status: 500 Internal Server Error
ERROR: Service 'fidesops' failed to build : Build failed
make: *** [compose-build] Error 1

eastandwestwind · 2022-06-13T14:24:08Z

Aside from my comment above ^ the code itself looks good.

Not blocking, but would like to add documentation for how to nav to the admin ui when the fidesops server is up and running.

TheAndrewJackson · 2022-06-13T14:53:28Z

@TheAndrewJackson I'm getting this build err locally when trying to make server, any ideas?

Building fidesops
[+] Building 1.2s (6/6) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                   0.0s
 => => transferring dockerfile: 2.08kB                                                                                                                                                                 0.0s
 => [internal] load .dockerignore                                                                                                                                                                      0.0s
 => => transferring context: 208B                                                                                                                                                                      0.0s
 => ERROR [internal] load metadata for docker.io/library/python:3.9.13-slim-buster                                                                                                                     1.1s
 => ERROR [internal] load metadata for docker.io/library/node:16                                                                                                                                       1.1s
 => [auth] library/python:pull token for registry-1.docker.io                                                                                                                                          0.0s
 => [auth] library/node:pull token for registry-1.docker.io                                                                                                                                            0.0s
------
 > [internal] load metadata for docker.io/library/python:3.9.13-slim-buster:
------
------
 > [internal] load metadata for docker.io/library/node:16:
------
failed to solve with frontend dockerfile.v0: failed to create LLB definition: failed to authorize: rpc error: code = Unknown desc = failed to fetch oauth token: unexpected status: 500 Internal Server Error
ERROR: Service 'fidesops' failed to build : Build failed
make: *** [compose-build] Error 1

I ran into that issue this morning and resolved it by restarting my computer. I'm not completely sure what was wrong 🤷‍♂️

TheAndrewJackson · 2022-06-13T14:54:42Z

Aside from my comment above ^ the code itself looks good.

Not blocking, but would like to add documentation for how to nav to the admin ui when the fidesops server is up and running.

Good idea. Can we get that in a follow up docs ticket? @eastandwestwind

eastandwestwind · 2022-06-14T02:15:08Z

I was able to resolve the make server issues, and got the app running.

Main question is, when I nav to http://0.0.0.0:8080/static/index.html, i'm receiving the placeholder index.html instead of the index.html supposedly copied over from the build in Dockerfile here. Is this expected?

eastandwestwind · 2022-06-14T02:20:48Z

I've added a separate issue for docs here- #644

TheAndrewJackson · 2022-06-14T13:09:50Z

I was able to resolve the make server issues, and got the app running.

Main question is, when I nav to http://0.0.0.0:8080/static/index.html, i'm receiving the placeholder index.html instead of the index.html supposedly copied over from the build in Dockerfile here. Is this expected?

Ahh, it's sort of expected. I ran into this a few times while developing this feature. What you're most likely seeing is the docker-compose.yml file is removing your UI files. Here is the snippet that is causing the issue:

    volumes:
      - type: bind
        source: ./
        target: /fidesops
        read_only: False
      - /fidesops/src/fidesops.egg-info

What's happening is the UI is built when the docker image is created initially. Then when make server or docker-compose up is run it mounts the local fidesops directory which doesn't contain the src/fidesops/build/static path so it creates the placeholder. If you comment out the snippet above in the docker-compose.yml file it should serve the UI like you're expecting as long as the initial docker image has the UI built into it.

eastandwestwind · 2022-06-14T14:15:38Z

Thanks for the clarification @TheAndrewJackson !

NevilleS · 2022-06-14T15:11:18Z

src/fidesops/main.py

+        if not WEBAPP_INDEX.is_file():
+            WEBAPP_DIRECTORY.mkdir(parents=True, exist_ok=True)
+            with open(WEBAPP_DIRECTORY / "index.html", "w") as index_file:
+                index_file.write("<h1>Privacy is a Human Right!</h1>")


I'm curious why is this needed? Is this just to handle a case where the docker build hasn't successfully built the admin UI and exported it, or is it more common than that?

I followed the same pattern that the fides used did for this update: https://github.com/ethyca/fides/blob/main/src/fidesapi/main.py#L59-L68. I believe that it is so there is always something to display in the event that the admin UI is toggled on.

It should probably be updated to display something for helpful to help the user know why the full admin ui isn't being displayed.

One common issue that Catherine, Sean, and I ran into while this was developed the the default config in the docker-compose.yml mounts a volume of the local fidesops files with this snippet

volumes: - type: bind source: ./ target: /fidesops read_only: False - /fidesops/src/fidesops.egg-info

Since the admin ui is only built while the docker image is created the ui files are missing from the local fidesops code causing the ui files to be missing once docker-compose up is ran.

This is okay because frontend development still uses npm run dev from clients/admin-ui and backend development won't typically need to use the admin ui from within the fidesops webserver. In the event that backend development needs the full admin within the webserver they can run the npm run prod-export command from within clients/admin-ui and that will build and move the full ui into the correct place within the backend code.

NevilleS · 2022-06-14T15:12:00Z

src/fidesops/core/config.py

@@ -128,6 +137,7 @@ class FidesopsConfig(FidesSettings):
    security: FidesopsSecuritySettings
    execution: ExecutionSettings
    root_user: RootUserSettings
+    admin_ui: AdminUiSettings


Can you add this to the log_all_config_values helper too? Sorry I didn't know a clean way to introspect the settings object to do this automatically

Yes, I'll make an issue for it now and update the code.

* Switch to Chakra image instead of nextjs * checking in progress towards static build * Mount static file directory * Update Dockerfile * Add dockerignore file * Fix file path * Update Dockerfile * Add admin ui config model * Conditionally build the UI * Fix env_prefix * Remove hello_world placeholder * Remove if logic from Dockerfile * Fix issues with BASE_URL * Refactor Head into component & update URNs * Fix import order * Update test URL path * Fix isort lint issue * Fix unit test failure * Update docs and changelog

TheAndrewJackson added 25 commits May 27, 2022 13:09

Switch to Chakra image instead of nextjs

33d6342

checking in progress towards static build

3c46d4c

Merge branch 'main' into deploy_ui_within_fidesops

c22ae1a

Merge branch 'main' into deploy_ui_within_fidesops

28c27a9

Mount static file directory

943fd5e

Update Dockerfile

29bacee

Add dockerignore file

eb0d3dd

Fix file path

0f72b88

Update Dockerfile

f0b0126

Merge branch 'main' into deploy_ui_within_fidesops

dd1a276

Merge branch 'main' into deploy_ui_within_fidesops

1136e18

Add admin ui config model

a07c9ba

Conditionally build the UI

3c67fc1

Fix env_prefix

0be87a4

Remove hello_world placeholder

d3c8205

Merge branch 'main' into deploy_ui_within_fidesops

6aeeea4

Remove if logic from Dockerfile

ef24232

Fix issues with BASE_URL

d9952d3

Refactor Head into component & update URNs

d03d003

Fix import order

f0985cf

Merge branch 'main' into deploy_ui_within_fidesops

1883d6b

Update test URL path

73ab9c6

Fix isort lint issue

18e7eb9

Fix unit test failure

2850309

Update docs and changelog

69da863

TheAndrewJackson marked this pull request as ready for review June 10, 2022 13:54

TheAndrewJackson requested a review from a team June 10, 2022 13:55

seanpreston assigned eastandwestwind Jun 10, 2022

TheAndrewJackson added 2 commits June 10, 2022 12:35

Merge branch 'main' into deploy_ui_within_fidesops

b257761

Merge branch 'main' into deploy_ui_within_fidesops

cf96321

conceptualshark reviewed Jun 10, 2022

View reviewed changes

Merge branch 'main' into deploy_ui_within_fidesops

b0c8722

TheAndrewJackson added 2 commits June 13, 2022 13:45

Merge branch 'main' into deploy_ui_within_fidesops

8ca57e3

Merge branch 'main' into deploy_ui_within_fidesops

a34f77c

eastandwestwind mentioned this pull request Jun 14, 2022

Add documentation on using admin-ui through fidesops server #644

Closed

eastandwestwind approved these changes Jun 14, 2022

View reviewed changes

eastandwestwind merged commit a3c4b02 into main Jun 14, 2022

eastandwestwind deleted the deploy_ui_within_fidesops branch June 14, 2022 14:16

NevilleS reviewed Jun 14, 2022

View reviewed changes

allisonking mentioned this pull request Jun 15, 2022

Add the UI to the compose deployment ethyca/fides#757

Merged

13 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deploy UI within fidesops #625

Deploy UI within fidesops #625

TheAndrewJackson commented Jun 9, 2022 •

edited

conceptualshark Jun 10, 2022

eastandwestwind commented Jun 13, 2022

eastandwestwind commented Jun 13, 2022

TheAndrewJackson commented Jun 13, 2022

TheAndrewJackson commented Jun 13, 2022

eastandwestwind commented Jun 14, 2022 •

edited

eastandwestwind commented Jun 14, 2022

TheAndrewJackson commented Jun 14, 2022

eastandwestwind commented Jun 14, 2022

NevilleS Jun 14, 2022

TheAndrewJackson Jun 14, 2022

NevilleS Jun 14, 2022

TheAndrewJackson Jun 14, 2022

		\| Admin UI Variables\|---\|---\|---\|---\|---\|
		\|`ENABLED` \| `FIDESOPS__ADMIN_UI__ENABLED` \| bool \| False \| True \| Toggle whether the `/static` file directory is mounted to serve the Admin UI

Deploy UI within fidesops #625

Deploy UI within fidesops #625

Conversation

TheAndrewJackson commented Jun 9, 2022 • edited

Purpose

Changes

Checklist

Ticket

conceptualshark Jun 10, 2022

Choose a reason for hiding this comment

eastandwestwind commented Jun 13, 2022

eastandwestwind commented Jun 13, 2022

TheAndrewJackson commented Jun 13, 2022

TheAndrewJackson commented Jun 13, 2022

eastandwestwind commented Jun 14, 2022 • edited

eastandwestwind commented Jun 14, 2022

TheAndrewJackson commented Jun 14, 2022

eastandwestwind commented Jun 14, 2022

NevilleS Jun 14, 2022

Choose a reason for hiding this comment

TheAndrewJackson Jun 14, 2022

Choose a reason for hiding this comment

NevilleS Jun 14, 2022

Choose a reason for hiding this comment

TheAndrewJackson Jun 14, 2022

Choose a reason for hiding this comment

TheAndrewJackson commented Jun 9, 2022 •

edited

eastandwestwind commented Jun 14, 2022 •

edited