Adding a Makefile and basic tests.

[metcalfc]

This approach uses bats to create a generic docker-compose test. It
checks that images are pullable, projects are buildable, up works, and
if ports are exposed that something is listening.

The tests can be tailored for each example. As long as they are the same
though, you can edit lib/test.bats.example and then make update-tests
and all the tests will be synced.

This adds three project dependencies: Makefile, BATS, and nc. A full clean
test run is PAINFULLY long (~14 minutes on my desktop) mostly
dominated by the build steps.

I've also added a target to just test whats changed. Since most of the time
PRs will just change one project. By default git finds the directories with
changes compared to origin master. You can override this with
BASE_REF=<whatever> make test-changed.

[glours]

Hi @metcalfc
Can't we use the Docker image of bats to do that instead of adding the whole bats source code in the lib directory?
Because all the test files are identical, it should be interesting to have only 1 test file and to loop over directories to execute tests.
In that way we'll only have 1 test file to maintain and we'll sure that a new sample will be automatically covered by test

[metcalfc]

The Dockerfile approach would require a docker in docker setup. I didnt tackle that out of the gate because its a PITA. If this approach works for folks we could add it later. The vendored code in lib are libraries for bats. You still have to install bats locally or run the docker image. I thought about the single test but thought eventually we would tailor the tests to each project. The current generic test is useful to start. Eventually though we could add tests for expected output, http status codes (vs generic nc), test api endpoints, etc. At this point they’d diverge. The generic could be enough really depends on what level of fidelity we want. This wont catch changes that break the app but leave the app in a running state which seems like a miss. Also bats is pretty opinionated about where the tests are defined and run from. For example its not possible to generate tests on the fly (or at least i couldnt make it work). Initially I tried to generate individual port tests but the way bats works with multiple passes for each test that doesnt work because the tests get renamed as they are generated and bats files are “run” once for every test. Bats is sort of magic and its not as simple as its just a script. Folks could decide that bats isnt the route and instead we should for example take a node/npm dependency and write a library to shell out, etc that just seemed like a lot of work vs bash.

…

On Aug 10, 2020, at 6:40 AM, Guillaume Lours ***@***.***> wrote:  @glours commented on this pull request. Hi @metcalfc Can't we use the Docker image of bats to do that instead of adding the whole bats source code in the lib directory? Because all the test files are identical, it should be interesting to have only 1 test file and to loop over directories to execute tests. In that way we'll only have 1 test file to maintain and we'll sure that a new sample will be automatically covered by test — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[metcalfc]

Over a 🥃 I spent some time trying to put all this in docker. I wanted to see if I could get away without full dind. The dockerfile:

FROM docker:19.03

ARG UID
ARG GID

RUN apk update && apk add bash bash-completion gawk make netcat-openbsd docker-compose findutils git curl && \
    rm -rf /var/lib/apt/lists/* && \
    rm /var/cache/apk/* && \
    addgroup -S -g $GID awesome && adduser -S -G awesome -u $UID awesome && \
    ln -s /opt/bats/bin/bats /usr/local/bin/bats && \
    git clone --depth 1 --branch v1.2.1 https://github.com/bats-core/bats-core.git /opt/bats

COPY bashrc /home/awesome/.bashrc

USER awesome

VOLUME  /code
WORKDIR /code

ENTRYPOINT ["bash"]

A couple of Makefile rules:

UID := $(shell id -u)
GID := $(shell grep docker /etc/group| cut -d: -f3)

docker:
	docker build --build-arg=UID=${UID} --build-arg=GID=${GID} -t awesome-compose ./extras

shell:
	docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock -v ${PWD}:/code awesome-compose

Gets you part way there. Unfortunately when it comes to testing the network ports the external network isn't available. So all those checks will fail. Unless I've missed something you would need a full dind setup. I can play around with it more when I get back from vacation.

[metcalfc]

@glours Walking the 🐕 with a ☕️ and network=host popped into my head. Not sure where it was last night. Anyway, took a pass at containing the toolchain. There is more work to do. I took the easy route and apk add docker-compose which gets a release or two ago. We should parameterize Docker and Compose to get exact versions. But its a start. This patch is... large now. I can squash it and break it up if we need to. I'll be back next week (2020/08/17).

# build the image. Since everyone's GID for docker is going to be different (Mac, Linux, WSL2, etc) there isn't 
# a point in pushing an image to a repo. Just build a local toolchain image matching the users UID/GID
make docker 

# interactive shell with all the tools (awk, nc, etc)
make shell

# the tests
make docker-test-all
make docker-test-changed

[glours]

Awesome thanks @metcalfc 👍
I'll take a look in detail during your holidays, I try to see if we can simplify some parts or how we can split it to integrate your changes easily