E2E Integration test environment setup

[pavish]

Fixes #639

Initial approach:

There are a few things to note:

• Playwright does not support linux arm64 images yet. It means that our integration tests will not work within docker in Mac M1 processors. Playwright team has mentioned that this will be fixed with their next major release.
• Playwright does not work with debian, mainly because the browser packages do not. Their official support is only for ubuntu. I tried modifying our docker setup to use ubuntu as the base, and install python, node and playwright over it. This lead to docker builds being super slow.
• I created a base image with ubuntu + python3.9 + node14 + playwright, and published it to dockerhub. I've used this image as our base image, so that our builds are faster.
• This image is currently in my account (did this to test it out) and with a free tier in dockerhub (only has limited pulls per day - around 200 for 6 hours): https://hub.docker.com/r/pavish73/mathesar-base/tags.
• I wanted to check if this approach sounds good to all of you. If it does, we can create an account for mathesar in dockerhub, and apply for their open source program: https://www.docker.com/community/open-source/application. This will give us unlimited pulls per day.
• The dockerfiles used to create the base imags can be placed in a dedicated repo.

Update on approach:

• Refer comment E2E Integration test environment setup #746 (comment)

Checklist

• My pull request has a descriptive title (not a vague title like Update index.md).
• My pull request targets the master branch of the repository
• My commit messages follow best practices.
• My code follows the established code style of the repository.
• I added tests for the changes I made (if applicable).
• I added or updated documentation (if applicable).
• I tried running the project locally and verified that there are no
  visible errors.

Developer Certificate of Origin

Developer Certificate of Origin

Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.


Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

[seancolsen]

I tested this locally and it works for me.

Without more Docker expertise I'm not able to comment on the overall approach.

Just one small documentation change to suggest:

• In README.md, change one instance of mathesar_db_1 to mathesar_db.

[mathemancer]

I think we should try really hard to sync the environment in the docker images with the environment running on staging. I'm okay with trying to move staging over to Ubuntu, though. At this point, it's a pretty common server OS. OTOH, that might be painful to implement at the moment (not sure about that part). What versions of Debian did you check?

Edit: As for the approach, I think we'd want to automate building for publication and publishing in order to make updating and changing things easier. Other than that, it makes sense to me.

[pavish]

@mathemancer

As for the approach, I think we'd want to automate building for publication and publishing in order to make updating and changing things easier.

We can maintain the docker files used to create the images for mathesar-base in a sub folder, and automate publishing the images. We will need to create an account for mathesar on docker-hub.

These are the contents of the docker files used to create pavish73/mathesar-base images.

amd64/arm64 image. tag: latest

FROM buildpack-deps:focal

ARG DEBIAN_FRONTEND=noninteractive

# Install python
# ensure local python is preferred over distribution python
ENV PATH /usr/local/bin:$PATH

# http://bugs.python.org/issue19846
# > At the moment, setting "LANG=C" on a Linux system *fundamentally breaks Python 3*, and that's not OK.
ENV LANG C.UTF-8

# Download node14 source
RUN curl -fsSL https://deb.nodesource.com/setup_14.x | bash -

# extra dependencies (over what buildpack-deps already includes)
RUN apt-get update && apt-get install -y --no-install-recommends \
		libbluetooth-dev \
		tk-dev \
		uuid-dev \
        sudo \
        nodejs \
	&& rm -rf /var/lib/apt/lists/*

ENV GPG_KEY E3FF2839C048B25C084DEBE9B26995E310250568
ENV PYTHON_VERSION 3.9.7

RUN set -ex \
	\
	&& wget -O python.tar.xz "https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz" \
	&& wget -O python.tar.xz.asc "https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz.asc" \
	&& export GNUPGHOME="$(mktemp -d)" \
	&& gpg --batch --keyserver hkps://keys.openpgp.org --recv-keys "$GPG_KEY" \
	&& gpg --batch --verify python.tar.xz.asc python.tar.xz \
	&& { command -v gpgconf > /dev/null && gpgconf --kill all || :; } \
	&& rm -rf "$GNUPGHOME" python.tar.xz.asc \
	&& mkdir -p /usr/src/python \
	&& tar -xJC /usr/src/python --strip-components=1 -f python.tar.xz \
	&& rm python.tar.xz \
	\
	&& cd /usr/src/python \
	&& gnuArch="$(dpkg-architecture --query DEB_BUILD_GNU_TYPE)" \
	&& ./configure \
		--build="$gnuArch" \
		--enable-loadable-sqlite-extensions \
		--enable-optimizations \
		--enable-option-checking=fatal \
		--enable-shared \
		--with-system-expat \
		--with-system-ffi \
		--without-ensurepip \
	&& make -j "$(nproc)" \
	&& make install \
	&& rm -rf /usr/src/python \
	\
	&& find /usr/local -depth \
		\( \
			\( -type d -a \( -name test -o -name tests -o -name idle_test \) \) \
			-o \( -type f -a \( -name '*.pyc' -o -name '*.pyo' -o -name '*.a' \) \) \
		\) -exec rm -rf '{}' + \
	\
	&& ldconfig \
	\
	&& python3 --version

# make some useful symlinks that are expected to exist
RUN cd /usr/local/bin \
	&& ln -s idle3 idle \
	&& ln -s pydoc3 pydoc \
	&& ln -s python3 python \
	&& ln -s python3-config python-config

# if this is called "PIP_VERSION", pip explodes with "ValueError: invalid truth value '<VERSION>'"
ENV PYTHON_PIP_VERSION 21.2.4
# https://github.com/docker-library/python/issues/365
ENV PYTHON_SETUPTOOLS_VERSION 57.5.0
# https://github.com/pypa/get-pip
ENV PYTHON_GET_PIP_URL https://github.com/pypa/get-pip/raw/d781367b97acf0ece7e9e304bf281e99b618bf10/public/get-pip.py
ENV PYTHON_GET_PIP_SHA256 01249aa3e58ffb3e1686b7141b4e9aac4d398ef4ac3012ed9dff8dd9f685ffe0

RUN set -ex; \
	\
	wget -O get-pip.py "$PYTHON_GET_PIP_URL"; \
	echo "$PYTHON_GET_PIP_SHA256 *get-pip.py" | sha256sum --check --strict -; \
	\
	python get-pip.py \
		--disable-pip-version-check \
		--no-cache-dir \
		"pip==$PYTHON_PIP_VERSION" \
		"setuptools==$PYTHON_SETUPTOOLS_VERSION" \
	; \
	pip --version; \
	\
	find /usr/local -depth \
		\( \
			\( -type d -a \( -name test -o -name tests -o -name idle_test \) \) \
			-o \
			\( -type f -a \( -name '*.pyc' -o -name '*.pyo' \) \) \
		\) -exec rm -rf '{}' +; \
	rm -f get-pip.py

ENV PYTHONUNBUFFERED=1
ENV DOCKERIZE_VERSION v0.6.1

# Install dockerize
RUN wget https://github.com/jwilder/dockerize/releases/download/$DOCKERIZE_VERSION/dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz \
    && tar -C /usr/local/bin -xzvf dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz \
    && rm dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz

CMD ["/bin/bash"]

amd64 image, tag: latest-all (contains playwright)

FROM pavish73/mathesar-base:latest

RUN pip install playwright==1.15.3
RUN pip install pytest-playwright==0.2.1
RUN playwright install
RUN playwright install-deps

CMD ["/bin/bash"]

Using these directly would make building local images very slow. My suggestion is only to publish an intermediary image.

I think we should try really hard to sync the environment in the docker images with the environment running on staging

I don't think this would be hard. We only have to replace the base image used in staging with our new base image. Everything else should work out of the box. Although, we would have to test it out and confirm.

What versions of Debian did you check?

Buster and Bullseye

[pavish]

@seancolsen

Just one small documentation change to suggest: In README.md, change one instance of mathesar_db_1 to mathesar_db.

I've merged with the latest master in which this is handled.

[pavish]

Integ tests seem to be failing after I merged with master which only had minor updates to the dockerfile. I'll take a look into this.

[pavish]

@mathemancer @kgodey I'm running to an issue which requires your help. I'm not sure how the pipeline passed earlier.

I've updated our workflow to record videos, which can be seen in the workflow summary (https://github.com/centerofci/mathesar/actions/runs/1382112450).

The error thrown is 'database "mathesar" does not exist', which I guess is fine. We have setup.cfg which mentions mathesar_db_test to be the database, however while accessing the UI, it still refers the mathesar db.

Edit: I think it might be because the application reads from the .env file, and only pytest has different value for db, from setup.cfg file. I guess updating the .env file on the integration test environment should suffice. Let me know your thoughts.

Edit2: If the application reads from the .env file, mathesar db should have been created (since I have run migrations on the workflow). So, not sure what's happening here.

[mathemancer]

Edit2: If the application reads from the .env file, mathesar db should have been created (since I have run migrations on the workflow). So, not sure what's happening here.

The migrations don't create the mathesar database, the install.py script does. But, I see you've run that as well. Note that the mathesar database is not owned by the webservice.

[pavish]

@mathemancer

The migrations don't create the mathesar database, the install.py script does. But, I see you've run that as well.. Note that the mathesar database is not owned by the webservice.

I understand that the install.py script creates the db, which it reads from the .env file. I did run the script, but our service throws an error stating that the db mathesar is not present. It works fine locally but not on GH, I wanted to check if I missed something on the GH action workflow.

[mathemancer]

see the line comment.

[mathemancer]

I think the usual way is for the hostname to be the header of the docker-compose.yml file section (so db in this case). I wonder if the version of docker you (we) have locally handles that smoothly, but the version on Github doesn't. Maybe try reverting this change.

[pavish]

It was db but after the workflow failed, I updated it to mathesar_db to check if that'll make a change. The result was the same. I could try changing it back again.

[kgodey]

I missed this from @mathemancer when reading through the conversation previously.

I think we should try really hard to sync the environment in the docker images with the environment running on staging. I'm okay with trying to move staging over to Ubuntu, though.

I don't think we should. I made a deliberate decision to not use Docker in our staging setup so that we'd be deploying (and testing) Mathesar in two different environments on a regular basis. I think it makes sense to keep staging on Debian and Docker-free.

[pavish]

@kgodey @mathemancer

I have committed the Dockerfile directly for E2E tests.

This currently takes ~17 minutes to run on the workflow, and a lot more time locally. The other approaches require more thought. I also tried to simplify the docker setup by using binaries instead of compiling from source (for python and playwright) but that lead to different issues. I didn't want to go down that rabbit hole again and spend any more time on this PR.

This has a number of changes and I'm dismissing the approval to avoid accidental merge without further review.

[kgodey]

@pavish Looks good. Please fix the issue with the issue label workflow and merge.

[kgodey]

@pavish This change seems like a bad merge.

[pavish]

@kgodey This was added in 12ef48d, which was mentioned here as harmless: #746 (comment)

There wasn't another step with the same name. Should I go ahead and remove this?

[kgodey]

@pavish it was superseded by 24e615a. I thought I had removed the older one. Please make sure that there's only one and it's the latest one (it should be setting the status to Waiting in the last line, not Blocked).

[kgodey]

I think I got confused and forgot that I added it in this PR.

[pavish]

Thanks. Updated in e5bfc70