reuse built frontend in ci, merge compose files #6674

AndrewChubatiuk · 2023-12-24T10:26:23Z

What type of PR is this?

Description

renamed docker-compose.yml to compose.yml as it's supported by docker compose v2 by default
merged all compose files to simplify local backend and e2e tests execution. added profiles:
- local - spins up all services in a setup, that previously was in docker-compose.yml
- e2e - spins up all services in a setup, that previously was in .ci/docker-compose.cypress.yml
- default(no profile set) - spins up all services in a setup, that previously was in .ci/docker-compose.yml
added ability to build frontend inside docker or reuse prebuilt frontend locally for CI purposes to speed up build
added arm64 build

How is this tested?

Unit tests (pytest, jest)
E2E Tests (Cypress)
Manually - executed e2e and unit tests locally against all three compose profiles
N/A

Mobile & Desktop Screenshots/Recordings (if there are UI changes)

AndrewChubatiuk · 2024-01-11T23:41:01Z

@justinclift could you please review this MR?

justinclift · 2024-01-12T00:07:28Z

@AndrewChubatiuk It's unlikely I'll have much free time to review Redash stuff today. 😦

Maybe one of the others can help instead? 😄

codecov · 2024-01-16T05:41:29Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 63.37%. Comparing base (af0773c) to head (15b94b9).
Report is 2 commits behind head on master.

❗ Current head 15b94b9 differs from pull request most recent head 1619b0b. Consider uploading reports for the commit 1619b0b to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6674      +/-   ##
==========================================
- Coverage   63.82%   63.37%   -0.45%     
==========================================
  Files         161      162       +1     
  Lines       13060    13170     +110     
  Branches     1803     1819      +16     
==========================================
+ Hits         8335     8347      +12     
- Misses       4425     4532     +107     
+ Partials      300      291       -9

see 21 files with indirect coverage changes

justinclift · 2024-01-18T01:16:51Z

Thinking this bit over:

renamed docker-compose.yml to compose.yml as it's supported by docker compose v2 by default

I'm unsure if we want that rename. My thinking is that docker-compose.yml is widely known and recognised, while compose.yml isn't.

.github/workflows/ci.yml

justinclift · 2024-01-18T01:46:15Z

.github/workflows/ci.yml

+      REDASH_PRODUCTION: "true"
+      PERCY_TOKEN: ddbcffd4497c339ea4dd356b9cd04c998f84f311df06b3dc96146a8a1228d170
+      CYPRESS_PROJECT_ID: 924cka
+      CYPRESS_RECORD_KEY: c0592115-a05c-4746-a203-ffc080db6881


Same question as above. 😉

these values were just b64 encoded. need to move them to secrets

I can remove these values if you can move them to secrets

This sounded familiar, so after some looking I've found previous discussion about those values:

f3892e0#commitcomment-120444257

Looks like they were put there on purpose when we were previously using CircleCI.

... this was intentional for CircleCI, because secrets don't work with PRs from Forked repositories.

We've moved from CircleCI to GitHub Actions, but the same thing might apply. We'd probably want to double check though, just to be sure.

github actions with pull_request event type have the same behavior as in CircleCI, but we can try pull_request_target, which requires more attention to PR before running it, cause it provides a token with a write access to base repo, but also provides an access to base repo secrets

AndrewChubatiuk · 2024-01-30T21:12:05Z

@justinclift could you please review it again?

AndrewChubatiuk · 2024-02-03T22:26:35Z

@guidopetri @justinclift @eradman could you please review this MR?
With pull_request_target event type (which redash was recently switched to) Github Actions uses a pipeline from a target(master) branch, so this MR will never be green

eradman · 2024-02-05T14:10:32Z

When I run the build there are a number of new warnings:

$ make
docker compose build
WARN[0000] The "COMMIT_INFO_SHA" variable is not set. Defaulting to a blank string.
WARN[0000] The "COMMIT_INFO_REMOTE" variable is not set. Defaulting to a blank string.
WARN[0000] The "CYPRESS_PROJECT_ID" variable is not set. Defaulting to a blank string.
WARN[0000] The "PERCY_TOKEN" variable is not set. Defaulting to a blank string.
WARN[0000] The "PERCY_BRANCH" variable is not set. Defaulting to a blank string.
WARN[0000] The "COMMIT_INFO_AUTHOR" variable is not set. Defaulting to a blank string.
WARN[0000] The "COMMIT_INFO_BRANCH" variable is not set. Defaulting to a blank string.
WARN[0000] The "COMMIT_INFO_MESSAGE" variable is not set. Defaulting to a blank string.
WARN[0000] The "CYPRESS_RECORD_KEY" variable is not set. Defaulting to a blank string.
WARN[0000] The "PERCY_COMMIT" variable is not set. Defaulting to a blank string.
WARN[0000] The "PERCY_PULL_REQUEST" variable is not set. Defaulting to a blank string.
WARN[0000] The "REDASH_PRODUCTION" variable is not set. Defaulting to a blank string.
WARN[0000] The "REDASH_PRODUCTION" variable is not set. Defaulting to a blank string.
WARN[0000] The "REDASH_PRODUCTION" variable is not set. Defaulting to a blank string.

The build works, but these messages make it look like I am missing a step. I'd say we should only emit a warning if the developer has an incomplete configuration.

AndrewChubatiuk · 2024-02-05T14:54:26Z

@eradman made these variable overridable and empty by default to remove warnings

bin/docker-entrypoint

compose.base.yaml

tests/__init__.py

redash/settings/__init__.py

justinclift · 2024-04-10T09:17:37Z

compose.yaml

@@ -68,3 +54,27 @@ services:
      - "1080:1080"
      - "1025:1025"
    restart: unless-stopped
+  cypress:
+    ipc: host


Interesting. Hadn't come across this docker setting before. Looks useful to know about for some situations. 😄

https://docs.docker.com/reference/cli/docker/container/run/#ipc

justinclift

This looks 95% good to me. The only two important things that I think need changing before it can be merged are:

Change the migrate name back to create_db in the docker entrypoint
- And whatever calls that name of course
Remove the change of REDASH_REDIS_URL environment variable
- It's already documented using this name
Remove the change of REDASH_DATABASE_URL environment variable
- It's already documented using this name

For updating this, please just add new commits so I don't have to re-review the entire thing. 😄

Makefile

justinclift

99% there now. Just needs the Makefile target changed in order to merge this. 😄

Makefile

justinclift

Looks good to me. 😄

justinclift · 2024-04-10T09:52:42Z

Ahhh, yeah. We might as well include the lint change too. All good. 😄

…)" Docker image node:18.1-bookworm does not resolve. After chaning to node:18-bookworm the resulting images do not match: ``` $ docker ps --format 'table {{.Image}}\t{{.Ports}}' IMAGE PORTS redash-server 0.0.0.0:5000->5000/tcp, :::5000->5000/tcp, 0.0.0.0:5678->5678/tcp, :::5678->5678/tcp redis:7-alpine 6379/tcp pgautoupgrade/pgautoupgrade:latest 0.0.0.0:15432->5432/tcp, :::15432->5432/tcp maildev/maildev 0.0.0.0:1025->1025/tcp, :::1025->1025/tcp, 0.0.0.0:1080->1080/tcp, :::1080->1080/tcp ``` Expected: ``` IMAGE PORTS redash-worker 5000/tcp redash-scheduler 5000/tcp redash-server 0.0.0.0:5678->5678/tcp, :::5678->5678/tcp, 0.0.0.0:5001->5000/tcp, :::5001->5000/tcp redis:7-alpine 6379/tcp maildev/maildev 0.0.0.0:1025->1025/tcp, :::1025->1025/tcp, 0.0.0.0:1080->1080/tcp, :::1080->1080/tcp pgautoupgrade/pgautoupgrade:latest 0.0.0.0:15432->5432/tcp, :::15432->5432/tcp ``` Reverted command used: ``` git reset --hard 24dec19 git reset --soft HEAD@{1} ```

…)" Docker image node:18.1-bookworm does not resolve. After chaning to node:18-bookworm the resulting images do not match: $ docker ps --format 'table {{.Image}}\t{{.Ports}}' IMAGE PORTS redash-server 0.0.0.0:5000->5000/tcp, :::5000->5000/tcp, 0.0.0.0:5678->5678/tcp, :::5678->5678/tcp redis:7-alpine 6379/tcp pgautoupgrade/pgautoupgrade:latest 0.0.0.0:15432->5432/tcp, :::15432->5432/tcp maildev/maildev 0.0.0.0:1025->1025/tcp, :::1025->1025/tcp, 0.0.0.0:1080->1080/tcp, :::1080->1080/tcp Expected: IMAGE PORTS redash-worker 5000/tcp redash-scheduler 5000/tcp redash-server 0.0.0.0:5678->5678/tcp, :::5678->5678/tcp, 0.0.0.0:5001->5000/tcp, :::5001->5000/tcp redis:7-alpine 6379/tcp maildev/maildev 0.0.0.0:1025->1025/tcp, :::1025->1025/tcp, 0.0.0.0:1080->1080/tcp, :::1080->1080/tcp pgautoupgrade/pgautoupgrade:latest 0.0.0.0:15432->5432/tcp, :::15432->5432/tcp Reverted command used: git reset --hard 24dec19 git reset --soft HEAD@{1}

…)" Revert all commits >= 3f19534 These changes were far to invasive to test properly. Reverted command used: git reset --hard 24dec19 git reset --soft HEAD@{1}

…)" Revert all commits >= 3f19534 These changes were far to invasive to test properly.

AndrewChubatiuk force-pushed the frontend-build branch from 9847808 to 130cc4b Compare December 24, 2023 10:30

AndrewChubatiuk mentioned this pull request Jan 4, 2024

Sqlalchemy 2.0 #6671

Closed

10 tasks

AndrewChubatiuk force-pushed the frontend-build branch 2 times, most recently from 5886fd2 to cc8ab2b Compare January 16, 2024 05:30

AndrewChubatiuk force-pushed the frontend-build branch 8 times, most recently from f0bb400 to 2f0f9ea Compare January 18, 2024 00:31

justinclift reviewed Jan 18, 2024

View reviewed changes

.github/workflows/ci.yml Outdated Show resolved Hide resolved

justinclift reviewed Jan 18, 2024

View reviewed changes

AndrewChubatiuk force-pushed the frontend-build branch 5 times, most recently from be7b32a to bc5ceba Compare January 18, 2024 12:19

AndrewChubatiuk force-pushed the frontend-build branch from 407ec87 to 2e7638e Compare February 5, 2024 09:04

AndrewChubatiuk force-pushed the frontend-build branch from 517ffdc to 6af65dc Compare February 5, 2024 14:53

AndrewChubatiuk force-pushed the frontend-build branch from f5274b1 to 3d39948 Compare February 10, 2024 07:36

AndrewChubatiuk mentioned this pull request Mar 27, 2024

removed custom configmap getredash/contrib-helm-chart#158

Merged

AndrewChubatiuk force-pushed the frontend-build branch 2 times, most recently from e3b319c to d3a51b7 Compare April 6, 2024 12:52

reuse built frontend in ci, merge compose files

2fe6376

AndrewChubatiuk force-pushed the frontend-build branch from d3a51b7 to 2fe6376 Compare April 9, 2024 19:30