Speed up ci (#383)

* Super broken commit of my throwing my brain at refactoring the Docker Compose config logic

* responsive mostly WIP

* began migrating to muiv5

* updated to MUI 5, finished first responsive pass

* cleanup

* fix integration tests

* quieter cypress

* add seo stuff

* Refactor CLI to be more modular

* got rid of unnecessary cypress step

* extend cypress first page load timeout

* try to catch cy bug

* mode in header'

* Switch to Yarn 2 and fix CLI Env bugs

* Update Yarn in gitignore

* Don't ignore nested yarn.lock files

* fix bugs that were related to the demo app running after migration to using yarn 3

* Fix bugs introduced, also compile static assets in local mode now

* Remove Lerna from CI scripts

* Attempt to fix CI again

* Add missing build scripts

* Add more type imports everywhere

* Cleanup versions and update Cypress

* Docs are still broken, but it is at least closer now!

* more permissive visit call

* fix local scripts

* Fix ESLint configuration in IntelliJ

* change urls to make sure that demo and tests are working. use xfvb-run
to bring up integration tests.

* fix docs by preventing hoisting of docs deps

* make lockfile deletion in CI also regenerate

* yarn install during no lockfile test

* trick yarn install about ci mode

* remove extra yarn install

* update lockfile

* lockfile hacks

* try CI=false

* set ci blank

* turn off fancy lockfile stuff

* remove go tests

* ignore linting

* cypress try global listener

* restore tmuxp and sign up for cypress dashboard

* reintroduced initial visit call to try to make cy tests work in CI

* more cypress headers

* get simple tokenizer working and restore go tests

* create a new workflow for checking evil deps

* simplify deps job

* simplify the RIGHT job

* simplify CI env var

* make sure buildkit is on inside cli container

* rewrite analytics inserter to support demo app

* make lint work again, update a ton of lint deps

* attempt to prevent lerna-bootstrap double run

* added some eslint ignores

* got all lints passing, definitely took on some tech-debt here

* improve bucket script

* fix lint

* fix lint bug with react-scripts and hopefully fix aws_resources path bug

* Another pass of the CI

* Move Golang tests into their own job

* Fix lint errors

* free changes made

* fix cypress verbose logging env var

* Split up the CI job into re-usable actions

* Rename jobs

* Fix what is likely a bad path

* Fix what is likely a bad path

* Remove action to test where bad syntax is

* Test if variable syntax is the problem

* Fix variable syntax

* Test uses syntax with version

* Another round of checks

* Another pass of tweaking

* Use underscores instead of dashes

* Fix invalid YAML

* Move actions around

* No nested actions

* Make input required

* Minimum test

* rename job

* Rename job

* id -> name

* id -> name

* Check variable syntax

* Double equals perhaps?

* Try with actual GitHub action script now

* Pass event through

* Remove cancel step

* Move steps back. maybe

* Move steps back. maybe

* Only call second script now

* Inline second workflow

* Pass through a shell

* Inline at the end

* Try inline again

* Bring back script

* Another test

* Just test merge branch check

* Test with an echo

* Test with a basic echo

* Remove conditional

* Smush everything together

* Remove echo line

* Remove conditional

* made cypress dashboard more smarter

* Remove all conditional logic by splitting up action

* Remove Yarn cache

* Cleanup

* update docusaurus to fix blogs and fix the awful looking admonition from beta 12

* make demo backend use pm2

* update yarn.lock

* Regenerate lockfile, fix CLI arguments for hosted demo

* Disable Nginx volume

* Fix a yarn error

* Fix yarn.lock

* Remove extraneous workdir statement

* Uncomment nginx

* Remove second build step from demo back end build

* Revert build change

* no fork pm2

* switch to pm2-runtime

* add mui types

* verbose start in CI

* yarn install

* made bucket script wait for file and brought back the use of a precache container

* switch cli workdir to repo

* try again in CI

* increase max mocked s3 body size in nginx for live demo

* fix nginx args

* fix admonitions

* yarn stuff

* include more install steps in the precache

* stop demo back end from starting prematurely

* delete &&, it wasnt needed

* rename integration test workflow to just test

Co-authored-by: Free Wortley <free@lunasec.io>
Co-authored-by: breadchris <chris@lunasec.io>

.dockerignore

@@ -2,8 +2,6 @@
 .idea/
 .github/
 
-docs/
-
 docker-compose.yaml
 docker-compose.*.yaml
 
@@ -12,4 +10,4 @@ outputs/
 **/build/
 **/node_modules/
 node_modules/
-.npmrc
+.npmrc

.editorconfig

@@ -0,0 +1,10 @@
+root = true
+
+[*]
+end_of_line = lf
+insert_final_newline = true
+
+[*.{js,json,yml}]
+charset = utf-8
+indent_style = space
+indent_size = 2

.eslintignore

@@ -4,4 +4,9 @@
 **/build
 **/generated
 js/demo-apps/packages/react-front-end/cypress/integration/secure_components_spec.ts
-**/ContactForm.jsx
+.pnp.cjs
+*.mjs
+deploy-apigateway-to-firehose.ts
+**/*.vue
+**/metrics-server-backend/*
+**/ContactForm.jsx

.eslintrc.js

@@ -21,10 +21,10 @@ module.exports = {
     node: true
   },
   extends: [
-    "plugin:vue/vue3-essential",
-    "@vue/typescript/recommended",
-    "@vue/prettier",
-    "@vue/prettier/@typescript-eslint",
+    // "plugin:vue/vue3-essential",
+    // "@vue/typescript/recommended",
+    // "@vue/prettier",
+    // "@vue/prettier/@typescript-eslint",
     'eslint:recommended',
     'plugin:@typescript-eslint/recommended',
     'plugin:@typescript-eslint/recommended-requiring-type-checking',
@@ -54,10 +54,12 @@ module.exports = {
   },
   plugins: [
     'react',
-    '@typescript-eslint'
+    '@typescript-eslint',
+    'jest'
   ],
   rules: {
-    'no-console': process.env.NODE_ENV === 'production' ? 'error' : 'off',
+    "@typescript-eslint/no-unsafe-argument": 1, // TODO: Re-enable this rule and fix all errors
+    'no-console': process.env.NODE_ENV === 'production' ? 'error' : 'off', // These never error, currently
     'no-debugger': process.env.NODE_ENV === 'production' ? 'error' : 'off',
     eqeqeq: 'error',
     quotes: ['error', 'single', { allowTemplateLiterals: true, avoidEscape: true }],

.github/ISSUE_TEMPLATE/bug_report.md

@@ -1,3 +1,16 @@
+<!--
+  ~ Copyright by LunaSec (owned by Refinery Labs, Inc)
+  ~
+  ~ Licensed under the Creative Commons Attribution-ShareAlike 4.0 International
+  ~ (the "License"); you may not use this file except in compliance with the
+  ~ License. You may obtain a copy of the License at
+  ~
+  ~ https://creativecommons.org/licenses/by-sa/4.0/legalcode
+  ~
+  ~ See the License for the specific language governing permissions and
+  ~ limitations under the License.
+  ~
+-->
 ---
 name: Bug report
 about: Create a report to help us improve

.github/workflows/check-deps.yaml

@@ -0,0 +1,23 @@
+name: Check Dependencies
+
+# This workflow checks to make sure that all dependency binaries are valid, to prevent a sneaky commit doing something malicious
+on:
+  pull_request:
+    branches: ['**', '**']
+  push:
+    branches: [master]
+
+jobs:
+  check-deps:
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v2
+    - uses: actions/setup-node@v2
+      with:
+        node-version: '16'
+
+    - name: dependencies
+      env:
+        IS_LUNASEC_CI: true
+      run: yarn install --immutable --immutable-cache --check-cache --inline-builds

.github/workflows/composite/merge-branch/action.yml

@@ -0,0 +1,22 @@
+name: 'Merge Master Branch'
+description: 'Merges the master Git branch into the current branch.'
+
+runs:
+  using: "composite"
+  steps:
+    # TODO: there may be a more idiomatic way to do this
+    - name: Check if we should skip this build
+      shell: bash
+      id: job-canceller
+      run: echo "::set-output name=cancelled::${{ github.event_name != 'pull_request' }}"
+
+     # Cancel the merge copy of this build(see the matrix above) if we are not in a PR
+    - name: cancelling
+      uses: andymckay/cancel-action@0.2
+      if: ${{ steps.job-canceller.outputs.cancelled == 'true' }}
+
+    # Merge with master (or whatever target branch) so we are actually testing what will happen after PR merges, not just this branch
+    - name: Merge target branch
+      shell: bash
+      run: git merge origin/${{ github.event.pull_request.base.ref }}
+      if: ${{ github.event_name == 'pull_request' }}

.github/workflows/composite/setup-docker-ci/action.yml

@@ -0,0 +1,40 @@
+name: 'Setup Docker CI Environment'
+description: 'Creates the LunaSec CI build environment'
+
+inputs:
+  merge_master:
+    description: 'If true, this merges the master branch during setup.'
+    required: true
+
+runs:
+  using: "composite"
+  steps:
+    - name: Set up Docker BuildKit
+      id: buildx
+      uses: docker/setup-buildx-action@v1
+      with:
+        install: true # sets buildx as the default for docker, which should apply to docker-compose commands
+        driver: docker
+
+    - uses: actions/setup-node@v2
+      with:
+        node-version: '16'
+
+    # specifically tag this bootstrap container to prevent rebuilds
+    - name: Build Lerna Bootstrap container
+      shell: bash
+      env:
+        # avoid warnings like "tput: No value for $TERM and no -T specified"
+        TERM: xterm
+        DOCKER_BUILDKIT: 1
+        COMPOSE_DOCKER_CLI_BUILD: 1
+      run: docker build --progress plain -f ./js/docker/demo.dockerfile -t lerna-bootstrap --target lerna-bootstrap .
+
+    - name: Build CLI Container
+      shell: bash
+      env:
+        # avoid warnings like "tput: No value for $TERM and no -T specified"
+        TERM: xterm
+        DOCKER_BUILDKIT: 1
+        COMPOSE_DOCKER_CLI_BUILD: 1
+      run: docker build --progress plain -f ./js/docker/demo.dockerfile -t repo_lunasec-cli --target lunasec-cli .

.github/workflows/documentation.yaml

@@ -41,19 +41,13 @@ jobs:
         with:
           node-version: '16'
 
-      # Speed up builds by caching node_modules
-      - uses: actions/cache@v2
-        with:
-          path: |
-            node_modules
-            */*/node_modules
-          key: ${{ runner.os }}-modules-${{ hashFiles('**/yarn.lock') }}
-
       - name: Lerna Bootstrap and Build Deploy Tool
+        env:
+          IS_LUNASEC_CI: true
+          CI: true
         run: |
-          npx lerna bootstrap --ci
+          yarn install --immutable --immutable-cache --inline-builds
           yarn compile:dev:infrastructure
-          npx lerna link
 
       - name: Build Repo
         working-directory: js/sdks

.github/workflows/integration.yaml

@@ -14,7 +14,7 @@
 #
 # This is a basic workflow to help you get started with Actions
 
-name: CI
+name: Tests
 
 # Controls when the workflow will run
 on:
@@ -36,53 +36,27 @@ jobs:
     env:
       # avoid warnings like "tput: No value for $TERM and no -T specified"
       TERM: xterm
-      RUNNING_IN_CI: true
       DOCKER_BUILDKIT: 1
       COMPOSE_DOCKER_CLI_BUILD: 1
     strategy:
       matrix:
         merge: ["merged", "not-merged"]
-        lockfile: ["lockfile","no-lockfile"]
+        lockfile: ["lockfile"] #,"no-lockfile"]
         exclude:
           - merge: "merged"
             lockfile: "no-lockfile"
 
     steps:
-
     - uses: actions/checkout@v2
       with:
         fetch-depth: 0
 
-    #TODO: there may be a more idiomatic way to do this
-    - name: Check if we should skip this build
-      id: job-canceller
-      run: echo "::set-output name=cancelled::${{ matrix.merge == 'merged' && github.event_name != 'pull_request'}}"
-
-    # Cancel the merge copy of this build(see the matrix above) if we are not in a PR
-    - name: cancelling
-      uses: andymckay/cancel-action@0.2
-      if: ${{ steps.job-canceller.outputs.cancelled == 'true' }}
-
-      # merge with master(or whatever target branch) so we are actually testing what will happen after PR merges, not just this branch
-    - name: Merge target branch
-      run: git merge origin/${{ github.event.pull_request.base.ref }}
-      if: ${{ matrix.merge == 'merged' && github.event_name == 'pull_request' }}
-
-    - uses: actions/setup-node@v2
-      with:
-        node-version: '14'
-        cache: yarn
-
-    - uses: actions/setup-go@v2
-      with:
-        go-version: '^1.17.1'
+    - name: Setup Branch
+      uses: ./.github/workflows/composite/merge-branch
+      if: ${{ matrix.merge == 'merged' }}
 
-    - name: Set up Docker BuildKit
-      id: buildx
-      uses: docker/setup-buildx-action@v1
-      with:
-        install: true # sets buildx as the default for docker, which should apply to docker-compose commands
-        driver: docker
+    - name: Setup Environment
+      uses: ./.github/workflows/composite/setup-docker-ci
 
     # Enable tmate debugging of manually-triggered workflows if the input option was provided
     - name: Setup tmate session
@@ -92,9 +66,13 @@ jobs:
       if: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.debug_enabled != 'false' }}
       timeout-minutes: 15
 
-    - name: Delete Lockfile
-      if: ${{ matrix.lockfile == 'no-lockfile' }}
-      run: rm yarn.lock
+    - uses: actions/setup-node@v2
+      with:
+        node-version: '16'
+
+#    - name: Delete and Regenerate Lockfile
+#      if: ${{ matrix.lockfile == 'no-lockfile' }}
+#      run: rm yarn.lock && CI="" yarn install #currently broken
 
 #    - uses: actions/setup-go@v2
 #      with:
@@ -116,22 +94,8 @@ jobs:
 #        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
 #      run: npm run license:check
 
-    # TODO: move this into docker, using the same container that the dedicated tokenizer uses
-    - name: Go Tests
-      run: yarn run test:unit:go
-
-    - name: Build CLI Container
-      run: docker build --progress plain -f ./js/docker/demo.dockerfile -t repo_lunasec-cli --target lunasec-cli .
-
-    # Hijack the cli image to quickly run linting and unit testing
-    - name: Lint
-      run: docker run --entrypoint yarn repo_lunasec-cli lint
-
-    - name: Unit Test
-      run: docker run --entrypoint yarn repo_lunasec-cli test:unit:js
-
     - name: Use CLI Container with docker.sock mounted to launch all other containers
-      run: docker run -v /var/run/docker.sock:/var/run/docker.sock -e HOST_MACHINE_PWD=$(pwd) repo_lunasec-cli start --no-sudo --local-build --env=tests
+      run: docker run -v /var/run/docker.sock:/var/run/docker.sock -e HOST_MACHINE_PWD=$(pwd) repo_lunasec-cli start --no-sudo --local-build --env=tests --verbose
 
     - name: docker logs
       if: ${{ always() && steps.job-canceller.outputs.cancelled == 'false' }}
@@ -158,4 +122,38 @@ jobs:
       env:
         SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}
         SLACK_TITLE: PR Build ${{ github.head_ref }} ${{ job.status }}
-        SLACK_COLOR: ${{ job.status }}
+        SLACK_COLOR: ${{ job.status }}
+
+  golang-unit-tests:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        merge: [ "merged", "not-merged" ]
+        lockfile: [ "lockfile" ] #,"no-lockfile"]
+        exclude:
+          - merge: "merged"
+            lockfile: "no-lockfile"
+    steps:
+      - uses: actions/checkout@v2
+        with:
+          fetch-depth: 0
+
+      - name: Setup Branch
+        uses: ./.github/workflows/composite/merge-branch
+        if: ${{ matrix.merge == 'merged' }}
+
+      - uses: actions/setup-go@v2
+        with:
+          go-version: '^1.17.1'
+
+      # TODO: move this into docker, using the same container that the dedicated tokenizer uses
+      - name: Go Tests
+        run: yarn run test:unit:go
+
+      - name: Slack Notification
+        uses: rtCamp/action-slack-notify@v2
+        if: ${{ steps.job-canceller.outputs.cancelled == 'false' }}
+        env:
+          SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}
+          SLACK_TITLE: PR Build ${{ github.head_ref }} ${{ job.status }}
+          SLACK_COLOR: ${{ job.status }}

.github/workflows/lint-code.yaml

@@ -0,0 +1,28 @@
+name: Lint Code plus JS Unit Tests
+
+# This workflow checks to make sure that all dependency binaries are valid, to prevent a sneaky commit doing something malicious
+on:
+  pull_request:
+    branches: ['**', '**']
+  push:
+    branches: [master]
+
+jobs:
+  lint-code:
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v2
+      with:
+        fetch-depth: 0
+
+    - name: Setup Environment
+      uses: ./.github/workflows/composite/setup-docker-ci
+
+    # Use the bootstrap image to quickly run linting and unit testing
+    - name: Lint
+      run: docker run --entrypoint yarn lerna-bootstrap lint
+
+    - name: Unit Test
+      run: docker run --entrypoint yarn lerna-bootstrap test:unit:js
+

.github/workflows/release.yml

@@ -51,10 +51,11 @@ jobs:
       - name: install lerna
         run: npm i -g lerna
 
-      - name: lerna bootstrap
+      - name: dependencies
         env:
           IS_LUNASEC_CI: true
-        run: lerna bootstrap --ci -- --frozen-lockfile
+          CI: true
+        run: yarn install --immutable --immutable-cache --check-cache --inline-builds
 
       - name: Configure CI Git User
         run: |
@@ -132,4 +133,4 @@ jobs:
       - name: git push new tags
         run: git push --tags
         env:
-          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}