…ed viz_scripts Dockerfile

viz_scripts Dockerfile contains ENV variables from docker-compose as well.

Added docker image commands to image_build_push yml for dashboard and notebook images

Added docker commands for dashboard and notebook  + Updated Dockerfile

1. Sed to jq change to make it consistent to what is being used in internal repos.

2. Renamed image pushed to docker hub.

Changed sed to jq + Renamed docker image in image_push

Currently the branch specified - "image-push-merge" is available locally on my system.

I use it to test the automated docker image push mechanism whenever any changes are merged to this branch.

Once, everything looks good, need to change this to master or main as per the repo.

Added TODO to change image push branch

Had added it initially for testing purposes.
Can remove now so it doesn't expose any sensitive info.

Removed printing Docker username

1. docker/cert.sh
- Uses new environment variable PROD_STAGE variable to determine whether in staging / production environment and only then install certificates, else skip.

2. Dockerfile
Added environment variables here, keeping the same default ENV values as the ones in docker-compose.yml.
Not adding in docker/Dockerfile.dev, since this change is being done primarily with the objective to aid with the automated build and push to Dockerhub.
This pushed image would then be the one that would be used in the internal environments as the base image, which would be based on the non-dev Dockerfile.

Added cert.sh + Modified Dockerfiles

The other ENV vars that were added in Docker-compose files are currently present in the Dockerfile in the external repo root.
Removing them from there and can add them as per requirement.
For local testing, add them when “docker run” is used using the -e flag.
For usage in stage / production, can be set by cloud team in AWS Codebuild as they currently do.

CRON_MODE
CRON_MODE can be moved to setting when docker run command is executed using -e flag.
This is because this is required in start_notebook.sh in CMD layer in Dockerfile which is executed when the container is run.
Hence, CRON_MODE value can be supplied later and not needed to be embedded in image.

PROD_STAGE
Not setting PROD_TRUE in docker run command since it is required during docker image build to decide whether to add certificates or not.
Hence, adding it in the Dockerfile so it’s available during docker build.

Removing ENV variables from Dockerfile

Can directly set DB_HOST since we can now use environment variables.
No need to use jq or sed to replace file contents and copy over files.

Removed sed / jq usage from start scripts

Created a new branch image-push-merge for e-mission-server in my forked repo.
Also modified image push workflow to build and push docker image to docker hub on push to image-push-merge branch.

Doing this since admin-dash was failing when was building from internal repo since this was still referring to old server code.
Changed Dockerfile and docker/Dockerfile.dev in public-dash and admin-dash to build from this new image
mukuflash03/e-mission-server:image-push-merge_2024-04-12--01-01

Redesigned server image is built from the image-push-merge branch on my personal forked repository.
This branch has a workflow run set up to build the docker image and push it to Dockerhub whenever a push or merge happens to image-push-merge branch.
Currently, I've been pushing to image-push and then creating a PR to merge into image-push-merge.

Doing this, so admin-dash and public-dash can build from the latest redesigned server code.
This is not the latest server code but the latest changes from my redesign PR.

Changing to build from base server image from my personal Dockerhub repository with redesigned server code.
Will need to change to build from Shankari's Dockerhub repository, once all changes are final.

Bumped up base server image tag

Bump up base server image tag

Added working code from join repo to fetch docker image tags using artifact download.

Bumped up after fixing "url" KeyError bug in this commit in server repo:
MukuFlash03/e-mission-server@e778b3f

Added push and workflow_dispatch trigger handling to fetch appropriate docker image tags.
Push trigger uses tag from artifact upload while workflow_dispatch uses tag from input parameters.

Check commit from admin-dash here:
MukuFlash03/op-admin-dashboard@39b289f

Updating Dockerfiles to use ARG environment variable with latest timestamp that will be passed through:
- `docker build --build-arg` command in Github actions in the workflow for automated pushing to Docker hub.
- `args: ` config field in docker-compose which will need to be set manually by developers locally.

Also, changing branch in fetch_runID and Dockerfiles to tags-combo-approach.

Public-dash was failing as I had incorrectly replaced the docker build and push commands with the commands used for admin-dash in its YAML file.

Public-dash has two images: frontend (dashboard) and viz_scripts (notebook-server) with their separate Dockerfiles.
Hence, was getting the error:
ERROR: failed to solve: failed to read dockerfile: open /var/lib/docker/tmp/buildkit-mount264844780/Dockerfile: no such file or directory

https://github.com/MukuFlash03/em-public-dashboard/actions/runs/8917300905/job/24490148189

For public-dash, admin-dash where ARGS are now being used, need to add the args under build command in the docker compose files.
Gives error if arg is at the same hierarchical level as build.

Also, public-dash docker-compose.yml (non-dev) version changed to have build: context, dockerfile ; unlike only build: frontend.
This allows adding args under build. Similar to how currently being built in docker-compose.dev.yml.

Also, args to be added under notebook-server and not dashboard since viz_scripts builds off of server image and not frontend, which is a node image.

Adding .env file which stores only docker image timestamp for the latest dockerhub e-mission-server image already pushed.

.env file overwritten in both types of trigger events - push and workflow_dispatch.

Added commit and push github actions as well for pushing latest changes to the .env file made via the workflow.

Lastly, docker-compose now also mentions the ENV variable name to be read from the .env file for the ARG value in the Dockerfile.

No changes required in the Dockerfiles.

Had to remove .env from the .gitignore file.

…-14-37

Reminder for things to change as per master branch of e-mission-server once changes are finalized.

Previous Push event triggers run failed

Error occurred in GitHub actions git add, commit, push step.
If file with no changes operated upon, it leaves an error:
“nothing to commit, working tree clean
Error: Process completed with exit code 1.”

Need to fix.

——

Quick fix is to make changes to .env file only if workflow_dispatch event is the trigger.
Don’t do anything for push event.
So, in case anyone modifies .env file on their own by using their own timestamp during testing, and pushes it as a part of their PR, then Shankari will have to ask them to revert the changes.
Else, their custom timestamp will make it to the repo code base.

Found something:
https://www.reddit.com/r/github/comments/ju3ipr/commit_from_github_action_only_when_changes_exist/

It should work but there’s a drawback of using “exit 0” - it will mask all errors generated during “git commit”.
This is bad and we won’t be able to see the reason why something wrong happened as the workflow would be shown as successful with a green tick.

Found a solution with git diff:
https://github.com/simonw/til/blob/main/github-actions/commit-if-file-changed.md

$ git diff --quiet || (git add README.md && git commit -m "Updated README")

However, I won’t be able to log any message saying that no changes to commit, tag not modified.
Hence, will possibly use just “git diff —quiet” with an if-else block.

Expected results:
- Push event triggers workflow.
- It writes DOCKER_IMAGE_TAG_1 fetched from last successful completed run to .env file.
- It sees that there is a difference in the latest committed .env file in the dashboard repo which includes older timestamp.
- Hence it runs git commit part of the script to reset to latest server timestamp.

Preparing the yml for merge with main.

Prepped for merge with main

Prepping for merge with main.

Moving a polished version of the work from the tag automation onto the main PR for the public dash

Per Shankari's permission, copying the cert file every time the viz_scripts dockerfile is run instead of deciding when to copy it based on an environment variable.

Removing cert.sh per Shankari's comments

Removing unnecessary change

Removing the `docker build` command from image_build_push.yml and replacing it with `docker compose`. Since tag names cannot be specified as an input in compose, the names of the tags have to be set in the docker-compose.yml file. Additionally, these tags have to be input as environment variables ahead of the `docker compose` part of the command.

I also renamed DOCKER_IMAGE_TAG to SERVER_IMAGE_TAG, since this is a more meaningful name. I will implement this change across the other repositories that use this variable name.

Removing unnecessary whitespace changes.

Adding a step to upload an artifact with the newly created image tag. This way, the file can be pulled into the internal repo with GH API.

Fallback unnecessary

I want to see if passing in the tags to the docker-compose in the way I'm trying to will actually work. We'll see!

Trying to test the workflow again.

Reverting the changes that I made to test the workflow with docker compose.