Skip to content

Commit

Permalink
fix: update docs
Browse files Browse the repository at this point in the history
  • Loading branch information
bkioshn committed May 3, 2024
1 parent f6269c6 commit 62e2c54
Show file tree
Hide file tree
Showing 4 changed files with 115 additions and 64 deletions.
46 changes: 24 additions & 22 deletions docs/src/guides/docs.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ To begin, clone the Catalyst CI repository:
git clone https://github.com/input-output-hk/catalyst-ci.git
```

Navigate to `docs` directory to find current documentation
Navigate to `docs` directory to find the current documentation
which you are reading right now.
This folder already has an `Earthfile` in it, which contains all build process.

Expand All @@ -40,19 +40,25 @@ This folder already has an `Earthfile` in it, which contains all build process.

```Earthfile
VERSION 0.8
IMPORT ../earthly/docs AS docs-ci
IMPORT .. AS cat-ci
IMPORT ../examples/postgresql AS postgresql-ci
# Copy all the source we need to build the docs
src:
# Common src setup
DO ../earthly/docs+SRC
DO docs-ci+SRC
# Now copy into that any artifacts we pull from the builds.
COPY --dir ../+repo-docs/repo includes
COPY --dir cat-ci+repo-docs/repo includes
# Copy docs we build in the postgres example.
COPY --dir postgresql-ci+build/docs
```

The first step of building process it preparing a source files.
The first step of building process is to prepare a source files.
It is mandatory to have a `src` directory with all documentation md files in it and `mkdocs.yml` file.
This directory and file will be picked during the execution of `+SRC` Function.
This directory and file will be picked during the execution of `+SRC` FUNCTION.
Also it is possible to replace defined `includes`, `macros` and `overrides` dirs
to customize some docs appearance and configuration.

Expand All @@ -75,35 +81,32 @@ Individual documentation targets can customize the theme in their `mkdocs.yml` f
docs:
FROM +src
DO ../earthly/docs+BUILD
DO docs-ci+BUILD
```

To build a docs artifact which will be used later just invoke `+BUILD` Function
on the already prepared docs environment `+src` target which we have discussed before.
To create a documentation artifact for future use, simply call the `+BUILD` FUNCTION within the existing documentation environment, specifically the `+src` target we previously discussed.

```Earthfile
# Make a docker image that can serve the docs for development purposes.
# This target is only for local developer use.
local:
DO ../earthly/docs+PACKAGE
DO docs-ci+PACKAGE
COPY +docs/ /usr/share/nginx/html
SAVE IMAGE cat-ci-docs:latest
```

To finally build a docker image which is pretty strait forward process,
you should firstly invoke `+PACKAGE` Function which will prepare an environment for future docs image,
next step is to copy builded artifact from the previous step to `/usr/share/nginx/html` folder.
And the last step is to save a docker image with the specified name, tag and registry if it is needed.

To create a Docker image, begin by executing the `+PACKAGE` FUNCTION to set up an environment for the future documentation image.
Then, copy the built artifact from the preceding step into the `/usr/share/nginx/html` directory. Finally, save the Docker image with the designated name, tag, and registry, if required.

## Local docs run

To locally run docs which it is needed to get a `earthly/docs/dev/local.py` python script
which will automatically invoke `+local` to build docs image what was discussed before.
This script will locally deploy docs and rebuild them if they changed every `10` seconds.
Script should be run from the root of the repository in which `docs` folder exists
with all documentation in it and already discussed `Earthfile`.
To run the documentation locally, the need of `earthly/docs/dev/local.py` Python script is required. This script automatically triggers the `+local` command to build the documentation image, as previously discussed.
It deploys the documentation locally and rebuilds it every `10` seconds if any changes are detected.
To execute the script, navigate to the root of the repository containing the docs folder with all the documentation and the previously discussed `Earthfile`.

Script arguments:

* `container_name` - Name of the container.
Expand All @@ -119,16 +122,15 @@ earthly/docs/dev/local.py cat-ci-docs:latest

<!-- markdownlint-disable max-one-sentence-per-line -->
!!! Note
To deploy docs for any other repositories as for example `catalyst-voices` or any other
as it was mentioned above it is needed to get `local.py` script and run it from the root
To deploy docs for any other repositories as for example `catalyst-voices`, it is needed to get `local.py` script and run it from the root
of your repo as for example `<path_to_catalyst_ci>/earthly/docs/dev/local.py <docker_image_name>`
<!-- markdownlint-enable max-one-sentence-per-line -->

## Doc's update PR

When a PR is raised the documentation for that PR is built and published.
When a PR is raised, the documentation for that PR is built and published.
Branch docs are published to `<pages>/branch/<branch_name>`.
`<branch_name>` is the name of the branch with all non alpha-numeric characters replaced by underscore (`_`).
`<branch_name>` is the name of the branch with all **non alpha-numeric characters replaced by underscore** (`_`).

When the branch is finally merged, the branch documentation is removed.
This allows us to easily validate what any PR will do to the published documentation before its published officially.
Expand Down
22 changes: 12 additions & 10 deletions docs/src/guides/languages/bash.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,18 +38,14 @@ Bash script checking is to be added to a repo that is already using Catalyst CI.
All that needs to happen is the following be added to the `Earthfile` in the root of the repo.

```Earthfile
# Internal: shell-check - test all bash files against our shell check rules.
shell-check:
FROM alpine:3.19
DO github.com/input-output-hk/catalyst-ci/earthly/bash:vx.y.z+SHELLCHECK --src=.
VERSION 0.8
IMPORT ./earthly/bash AS bash-ci
# check all repo wide checks are run from here
check:
# check-bash - test all bash files lint properly according to shellcheck.
check-bash:
FROM alpine:3.19
# Lint all bash files.
BUILD +shell-check
DO bash-ci+SHELLCHECK --src=.
```

Expand All @@ -71,11 +67,17 @@ These scripts are intended to be used inside Earthly builds, and not locally.
A common pattern to include these common scripts is the following:

```Earthfile
VERSION 0.8
IMPORT ../../utilities/scripts AS scripts
# Any appropriated target
builder:
# Copy our target specific scripts
COPY --dir scripts /scripts
# Copy our common scripts so we can use them inside the container.
DO ../../utilities/scripts+ADD_BASH_SCRIPTS
DO scripts+ADD_BASH_SCRIPTS
```

<!-- markdownlint-disable max-one-sentence-per-line -->
Expand Down
30 changes: 15 additions & 15 deletions docs/src/guides/languages/go.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,17 +48,18 @@ You can choose to either delete the file and start from scratch, or read the gui
```Earthfile
VERSION 0.8
IMPORT ../../earthly/go AS go-ci
deps:
# This target is used to install external Go dependencies.
FROM golang:1.21-alpine3.19
WORKDIR /work
# Any build dependencies should also be captured in this target.
RUN apk add --no-cache gcc musl-dev
# This Function automatically copies the go.mod and go.sum files and runs
# This FUNCTION automatically copies the go.mod and go.sum files and runs
# `go mod download` to install the dependencies.
DO ../../earthly/go+DEPS --ginkgo="false"
DO go-ci+DEPS --ginkgo="false"
```

The first target we are going to create will be responsible for downloading the external dependencies that our Go program uses.
Expand All @@ -68,18 +69,18 @@ dependencies.

We are going to be inheriting from an alpine image because we are pushing for a fully static build.
When using example as a starting place, feel free to change the base image.
This image is *only* used during the build steps, so it's not important we minimize its size.
This image is *only* used during the build steps, so it's not important to minimize its size.
Keep in mind, though, excessively large targets can impact the speed of the caching step (due to lots of I/O).

This target is also going to build responsible for installing external build dependencies.
This target is also going to be responsible for installing external build dependencies.
These are dependencies that are not specific to a language and usually get installed system-wide.
In our case, since we're building a static binary, we need `gcc` and `musl`.

Finally, the actual logic we will be using is encapsulate in a Function.
Finally, the actual logic we will be using is encapsulate in a FUNCTION.
This is a very common pattern, as an `Earthfile` can get repetitive across a repository.
In our case, we use the `go+DEPS` Function that will automatically copy our `go.mod` and `go.sum` files and then execute
In our case, we use the `go-ci+DEPS` FUNCTION that will automatically copy our `go.mod` and `go.sum` files and then execute
`go mod download`.
The Function will also establish a cache for the Go tooling.
The FUNCTION will also establish a cache for the Go tooling.
This means that, even if our source code changes, we'll see a substantial speed boost when compiling because the cache is preserved
across Earthly runs.

Expand All @@ -89,18 +90,17 @@ across Earthly runs.
src:
# This target copies the source code into the current build context
FROM +deps
COPY --dir cmd .
check:
# This target checks the overall health of the source code.
FROM +src
# This Function validates the code is formatted according to Go standards.
DO ../../earthly/go+FMT --src="go.mod go.sum cmd"
# This FUNCTION validates the code is formatted according to Go standards.
DO go-ci+FMT --src="go.mod go.sum cmd"
# This Function runs golangci-lint to check for common errors.
DO ../../earthly/go+LINT --src="go.mod go.sum cmd"
# This FUNCTION runs golangci-lint to check for common errors.
DO go-ci+LINT --src="go.mod go.sum cmd"
```

With our dependencies installed, we're now ready to start operating with the source code.
Expand All @@ -111,8 +111,8 @@ Any future targets which need access to the source code will inherit from this t

Now that the source code is available, we can begin performing static checks.
These checks are intended to verify the code is healthy and conforms to a certain standard.
As we did in the previous section, here we rely on Functions again to perform these checks.
These two Functions will validate the code formatting is correct and also perform a series of lints to validate code quality.
As we did in the previous section, here we rely on FUNCTIONs again to perform these checks.
These two FUNCTIONs, `go-ci+FMT` and `go-ci+LINT` will validate the code formatting is correct and also perform a series of lints to validate code quality.

Note that these checks are fast (compared to later steps) and perform quick feedback on code quality.
Since this is the first target run in CI, we want to fail the CI as quickly as possible if we can easily find code quality issues.
Expand Down
81 changes: 64 additions & 17 deletions docs/src/guides/languages/postgresql.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,39 +51,85 @@ You can choose to either delete the file and start from scratch, or read the gui
```Earthfile
VERSION 0.8
IMPORT ../../earthly/postgresql AS postgresql-ci
IMPORT ../../utilities/scripts AS scripts
IMPORT ../../ AS cat-ci
# Internal: builder is our Event db builder target. Prepares all necessary artifacts.
# CI target : dependency
builder:
FROM ./../../earthly/postgresql+postgres-base
DO postgresql-ci+BUILDER
# MUST Manually copy the .sqlfluff config used in the repo because the FUNCTION
# above can not be passed a reference to a local target as an argument.
COPY cat-ci+repo-config/repo/.sqlfluff .
```

<!-- TODO - FIX this part -->
Given `BUILDER`
```Earthfile
# Common build setup steps.
# Arguments:
# sqlfluff_cfg - REQUIRED - Location of repos .sqlfluff configuration file.
# migrations - OPTIONAL - Location of Migrations directory - DEFAULT: ./migrations
# seed - OPTIONAL - Location of Seed data directory - DEFAULT: ./seed
# refinery_toml - OPTIONAL - Location of refinery,toml which configures migrations. DEFAULT: ./refinery.toml
BUILDER:
FUNCTION
ARG migrations=./migrations
ARG seed=./seed
ARG refinery_toml=./refinery.toml
FROM +postgres-base
WORKDIR /build
COPY --dir ./migrations ./data ./refinery.toml .
DO ./../../earthly/postgresql+BUILDER
COPY --dir $sqlfluff_cfg .
COPY --dir $migrations .
COPY --dir $seed .
COPY --dir $refinery_toml .
```

The first target we are going to consider will be responsible to prepare a PostgreSQL environment (Earthly `+postgres-base` target),
migrations, migrations configuration and seed data (`COPY --dir ./migrations ./data ./refinery.toml .`),
doing some final build step (Earthly `+BUILDER` Function).
doing some final build step (Earthly `+BUILDER` FUNCTION).

In the next steps we are going to inheriting from this `+builder` target which contains all necessary data,
dependencies, environment to properly run PostgreSQL database.

### Running checks

```Earthfile
VERSION 0.8
IMPORT ../../earthly/postgresql AS postgresql-ci
IMPORT ../../utilities/scripts AS scripts
IMPORT ../../ AS cat-ci
# check if the sql files are properly formatted and pass lint quality checks.
# CI target : true
check:
FROM +builder
DO ./../../earthly/postgresql+CHECK
# Now you can run the script without the file not found error
DO postgresql-ci+CHECK
# Internal: builder is our Event db builder target. Prepares all necessary artifacts.
# CI target : dependency
builder:
DO postgresql-ci+BUILDER
build-sqlfluff:
BUILD ./../../earthly/postgresql+sqlfluff-image
# MUST Manually copy the .sqlfluff config used in the repo because the FUNCTION
# above can not be passed a reference to a local target as an argument.
COPY cat-ci+repo-config/repo/.sqlfluff .
# format all SQL files in the current project. Local developers tool.
# CI target : false
format:
LOCALLY
RUN earthly +build-sqlfluff
DO ./../../earthly/postgresql+FORMAT --src=$(echo ${PWD})
DO postgresql-ci+FORMAT --src=$(echo ${PWD})
```

With prepared environment and all data, we're now ready to start operating with the source code - `*.sql` files.
Expand All @@ -92,8 +138,8 @@ These checks are intended to verify the code is healthy and well formatted to a
and done with the help of the `sqlfluff` tool which is already configured during the `+postgres-base` target.

To apply and fix some formatting issues you can run `+format` target which will picks up directory
where your Earthly file lies in as a source dir for formatting and run `+FORMAT` Function.
Under the hood `+FORMAT` Function runs `sqlfluff-image` docker image,
where your Earthly file lies in as a source dir for formatting and run `+FORMAT` FUNCTION.
Under the hood `+FORMAT` FUNCTION runs `sqlfluff-image` docker image,
which contains the same configuration and setup which is applied during the `+check`.

<!-- markdownlint-disable max-one-sentence-per-line -->
Expand All @@ -105,20 +151,20 @@ which contains the same configuration and setup which is applied during the `+ch
### Build

```Earthfile
# build an event db docker image.
# CI target : true
build:
FROM +builder
ARG tag="latest"
ARG registry
DO ./../../earthly/postgresql+BUILD --image_name=example-db --tag=$tag --registry=$registry
DO postgresql-ci+BUILD --image_name=example-db
DO postgresql-ci+DOCS
```

With the `*.sql` files validation out of the way, we can finally build our PostgreSQL docker image.
Since we need migration and seed data files,
we'll inherit from the `builder` target.
The actual image build process is pretty straight-forward
and fully defined under the `+BUILD` Function.
and fully defined under the `+BUILD` FUNCTION.
The only thing it is needed to specify is a few arguments:

* `tag` - the tag of the image, default value `latest`.
Expand Down Expand Up @@ -186,6 +232,7 @@ otherwise will relies on remote PostgreSQL server connection
Finally we can test already configured and prepared PostgreSQL image
and trial it against 4 different cases

<!-- TODO - FIX this part? -->
```Earthfile
# Container runs PostgreSQL server, drops and initialise db, applies migrations, applies seed data.
test-1:
Expand Down

0 comments on commit 62e2c54

Please sign in to comment.