Bundle CyberChef into a container and publish to GCHR #1699

AshCorr · 2024-01-30T15:44:21Z

Bundle CyberChef into a container and publish to GCHR

Yet another attempt at bundling CyberChef into a container! There have been various attempts made before this one but they all seem to have been abandoned:

But as the famous saying goes "8th time's the charm" and maybe this PR will be the one that finally does it.

This PR adds a:

Containerfile ~~which uses simple-web-server a lightweight HTTP server built in rust for serving static files built in rust to serve CyberChef.~~
- EDIT: Now using Nginx, it results in a slightly larger image but with the benefit that we're using tried and tested software.
New steps to the release.yml workflow
- A step using docker-metadata-action which generates tags to be applied to the image from the release version (1, 1.0, 1.0.0 and latest)
- A step using buildah to build an OCI compliant container image. It builds a both an ARM64 image and a AMD64 image.
- And finally a step to push the built container to GHCR using the GITHUB_TOKEN built into github actions.
A step to the pull-requests.yml workflow to verify that the Containerfile builds correctly

and a couple of flyby fixes that can be dropped if needs be:

A version bump to ChromeDriver, fixing the failing UI tests being experience by all PRs. This was caused by a bump to Chrome in the ubuntu-latest runner. It looks like another user has also attempted this in Update chromedriver to v119 so that it is compatible with the chrome … #1656
Switch the Release workflow to use npm ci instead of npm install. npm ci respects the package-lock.json file whereas npm install will try and bump dependencies which is not ideal in a CI workflow as it adds uncertainty into which dependencies are being used in your production bundle.

Related to #1280

Why?

Theres various un-official images floating around of CyberChef, although I'm sure most of them are probably fine, but for a tool such as CyberChef where I'm inputting potential sensitive data I'd rather not take any chances.

Testing

I've published a version built on my own fork to ghcr.io/ashcorr/cyberchef:latest which may be used for testing.

podman run -p 8080:80 -it ghcr.io/ashcorr/cyberchef:latest or docker run -p 8080:80 -it ghcr.io/ashcorr/cyberchef:latest

CLAassistant · 2024-01-30T15:44:31Z

All committers have signed the CLA.

Containerfile

a3957273 · 2024-02-02T18:23:31Z

Fixed some merge conflicts from another PR that updated chromedriver.

AshCorr · 2024-02-02T19:06:27Z

Fixed some merge conflicts from another PR that updated chromedriver.

Thanks!

.github/workflows/pull_requests.yml

.github/workflows/releases.yml

.github/workflows/pull_requests.yml

a3957273

Just one minor change!

.github/workflows/releases.yml

a3957273 · 2024-02-08T22:47:35Z

When building this container locally, I notice it comes with a lot of context. Could we add a .dockerignore file so we transfer less other stuff? E.g.

node_modules
npm-debug.log
travis.log
build
.vscode
.idea
.*.swp
**/*.DS_Store
tests/browser/output/*
.node-version

SebastianThorn · 2024-02-09T07:39:14Z

This is relevant as well for this issue: #1582
Not sure if that is fixed or not, but for a production-build, there should be as few unnecessary components as possible.

a3957273 · 2024-02-09T15:49:25Z

@SebastianThorn A fair point. I'm inclined to merge this PR in even if it does require internet to compile, then accept other PRs to improve the process? A single PR doesn't necessarily need to be perfect, and I'm happy that a future improvement to allow offline building wouldn't impact existing users of this feature.

maxstanley · 2024-02-11T12:49:24Z

As you've reference my closed PR, I thought I would drop a comment and say this will be a great addition. Feel free to copy anything from my PR into this branch e.g. dockerignore.

I think for me as a user the main thing that is missing is an update to the README that demonstrates how to build with docker/podman and how an end-user might deploy this.

tyvm

maxstanley · 2024-02-11T14:51:17Z

README.md

+docker run -it -p 8080:80 ghcr.io/gchq/cyberchef:latest
+```
+
+This image is built and published through our [GitHub Workflows](.github/workflows/docker.yml)


Need to update the workflow file reference ;)

Oops, good catch!

maxstanley · 2024-02-11T16:53:31Z

.github/workflows/releases.yml

+      with:
+        images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
+        tags: |
+          type=semver,pattern={{major}}


Any reason for the omission of:

type=raw,value=latest

Its added automatically when using a type=semver pattern! See https://github.com/docker/metadata-action?tab=readme-ov-file#latest-tag

.github/workflows/releases.yml

SebastianThorn · 2024-02-12T07:02:46Z

The switch to Nginx is good.

…w too

…ecified

maxstanley · 2024-02-12T12:37:01Z

.github/workflows/pull_requests.yml

@@ -33,6 +33,20 @@ jobs:
      if: success()
      run: npx grunt prod

+    - name: Production Image Build
+      if: success()


Is this still required?

a3957273 · 2024-02-13T00:12:12Z

I'm going to merge this as is, because this conversation already has over three dozen comments. Happy to accept other PRs for improvements! 👍

Thanks so much for this @AshCorr. I love how you saw the other 7 PRs and decided to make an 8th 😛, persistence is the key!

a3957273 · 2024-02-13T00:31:33Z

@AshCorr Seems to be erroring on permissions, see: https://github.com/gchq/CyberChef/actions/runs/7879757258/job/21500616862

AshCorr · 2024-02-13T00:51:06Z

@AshCorr Seems to be erroring on permissions, see: https://github.com/gchq/CyberChef/actions/runs/7879757258/job/21500616862

Odd, I didn't touch that action thats erroring. I wonder if something wonky has happened where defining an explicit packages: write permission has caused default permissions to not be applied? My earlier statement of "Github permissions always confuse me" I think is relevant again 😅

By the looks of it you should be able to add an explicit contents: write permission to fix this, see svenstaro/upload-release-action#70 (comment)

AshCorr · 2024-02-13T00:55:24Z

Ah yes, that does appear to be the case as per https://docs.github.com/en/actions/security-guides/automatic-token-authentication#modifying-the-permissions-for-the-github_token

You can use the permissions key in your workflow file to modify permissions for the GITHUB_TOKEN for an entire workflow or for individual jobs. This allows you to configure the minimum required permissions for a workflow or job. When the permissions key is used, all unspecified permissions are set to no access, with the exception of the metadata scope, which always gets read access.

My apologies for missing that! I believe I commented out this action on my fork as I couldn't get it working, I assumed it was due to it being run on a fork, not due to lack of permissions.

a3957273 · 2024-02-13T00:56:50Z

No worries, super not clear. Removing all permissions and just using the defaults appear to have worked well!

Package now available here! https://github.com/gchq/CyberChef/pkgs/container/cyberchef

@AshCorr

AshCorr · 2024-02-13T01:11:21Z

No worries, super not clear. Removing all permissions and just using the defaults appear to have worked well!

Package now available here! https://github.com/gchq/CyberChef/pkgs/container/cyberchef

@AshCorr

Looks to be working! Thanks!

AshCorr commented Jan 30, 2024

View reviewed changes

Containerfile Outdated Show resolved Hide resolved

AshCorr marked this pull request as ready for review January 30, 2024 15:48

AshCorr force-pushed the ash/Containers! branch from 1390c8a to 6b222d9 Compare January 30, 2024 15:50

a3957273 requested changes Feb 2, 2024

View reviewed changes

.github/workflows/pull_requests.yml Show resolved Hide resolved

.github/workflows/releases.yml Outdated Show resolved Hide resolved

.github/workflows/releases.yml Show resolved Hide resolved

.github/workflows/releases.yml Show resolved Hide resolved

a3957273 reviewed Feb 3, 2024

View reviewed changes

.github/workflows/pull_requests.yml Outdated Show resolved Hide resolved

AshCorr force-pushed the ash/Containers! branch 2 times, most recently from 5e1686f to 73144c5 Compare February 8, 2024 10:14

AshCorr requested a review from a3957273 February 8, 2024 12:21

a3957273 requested changes Feb 8, 2024

View reviewed changes

.github/workflows/releases.yml Outdated Show resolved Hide resolved

maxstanley reviewed Feb 11, 2024

View reviewed changes

AshCorr force-pushed the ash/Containers! branch 2 times, most recently from 704ff8f to 879fd1f Compare February 12, 2024 10:46

AshCorr added 9 commits February 12, 2024 10:46

Add new steps to publish a containerized CyberChef to GHCR

0da3081

Build container on PR (but don't publish) to verify Containerfile syntax

3b26532

Generate prod build inside container

40a4872

Remove unecessary QEMU install step

7588e50

Remove unecessary if: success() from Github release workflow

dbdcb46

Add packages write permission

a072930

Increase nofile limit to 10,000 when building container on PR workflo…

4b95ab2

…w too

Remove ARM64 Container build from release workflow

2b85336

Swtich to nginx for container

1a2207a

AshCorr added 7 commits February 12, 2024 10:46

Switch to using Dockerfile

d700d1d

Add a comment to the README about using Docker

314b925

Add a .dockerignore file

abd9024

Fix CI using Containerfile

c5e8806

Remove image property from Container build & push actions

3f89a94

Fix PR workflow, Buildah requires atleast an image name if no tags sp…

7582abf

…ecified

Point to the correct workflow in the README for Container releases.

62dfa8f

AshCorr force-pushed the ash/Containers! branch from 879fd1f to 62dfa8f Compare February 12, 2024 10:46

maxstanley reviewed Feb 12, 2024

View reviewed changes

a3957273 merged commit 4b9d5a7 into gchq:master Feb 13, 2024
4 checks passed

a3957273 mentioned this pull request Feb 22, 2024

Add dockerfile and instructions #1215

Closed

oscrx mentioned this pull request Feb 22, 2024

Migrate to official cyberchef image bierteam/uber-kubernetes#428

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bundle CyberChef into a container and publish to GCHR #1699

Bundle CyberChef into a container and publish to GCHR #1699

AshCorr commented Jan 30, 2024 •

edited

CLAassistant commented Jan 30, 2024 •

edited

a3957273 commented Feb 2, 2024

AshCorr commented Feb 2, 2024

a3957273 left a comment

a3957273 commented Feb 8, 2024

SebastianThorn commented Feb 9, 2024

a3957273 commented Feb 9, 2024

maxstanley commented Feb 11, 2024 •

edited

maxstanley Feb 11, 2024

AshCorr Feb 12, 2024

maxstanley Feb 11, 2024

AshCorr Feb 12, 2024

SebastianThorn commented Feb 12, 2024

maxstanley Feb 12, 2024

a3957273 commented Feb 13, 2024

a3957273 commented Feb 13, 2024

AshCorr commented Feb 13, 2024 •

edited

AshCorr commented Feb 13, 2024 •

edited

a3957273 commented Feb 13, 2024

AshCorr commented Feb 13, 2024

Bundle CyberChef into a container and publish to GCHR #1699

Bundle CyberChef into a container and publish to GCHR #1699

Conversation

AshCorr commented Jan 30, 2024 • edited

Bundle CyberChef into a container and publish to GCHR

Why?

Testing

CLAassistant commented Jan 30, 2024 • edited

a3957273 commented Feb 2, 2024

AshCorr commented Feb 2, 2024

a3957273 left a comment

Choose a reason for hiding this comment

a3957273 commented Feb 8, 2024

SebastianThorn commented Feb 9, 2024

a3957273 commented Feb 9, 2024

maxstanley commented Feb 11, 2024 • edited

maxstanley Feb 11, 2024

Choose a reason for hiding this comment

AshCorr Feb 12, 2024

Choose a reason for hiding this comment

maxstanley Feb 11, 2024

Choose a reason for hiding this comment

AshCorr Feb 12, 2024

Choose a reason for hiding this comment

SebastianThorn commented Feb 12, 2024

maxstanley Feb 12, 2024

Choose a reason for hiding this comment

a3957273 commented Feb 13, 2024

a3957273 commented Feb 13, 2024

AshCorr commented Feb 13, 2024 • edited

AshCorr commented Feb 13, 2024 • edited

a3957273 commented Feb 13, 2024

AshCorr commented Feb 13, 2024

AshCorr commented Jan 30, 2024 •

edited

CLAassistant commented Jan 30, 2024 •

edited

maxstanley commented Feb 11, 2024 •

edited

AshCorr commented Feb 13, 2024 •

edited

AshCorr commented Feb 13, 2024 •

edited