Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) in gitpod/workspace-mysql

[christiansiewert]

Hi,

unfortunately we can't connect to our MySQL server when initializing our workspace while using gitpod/workspace-mysql docker image.

We have this in our .gitpod.Dockerfile:

FROM gitpod/workspace-mysql

USER gitpod

# Install php-dev
RUN sudo apt-get update -q \
    && sudo apt-get install -y php-dev

# Build and install Xdebug
RUN wget http://xdebug.org/files/xdebug-2.9.1.tgz \
    && tar -xvzf xdebug-2.9.1.tgz \
    && cd xdebug-2.9.1 \
    && phpize \
    && ./configure \
    && make \
    && sudo mkdir -p /usr/lib/php/20190902 \
    && sudo cp modules/xdebug.so /usr/lib/php/20190902 \
    && sudo bash -c "echo -e '\nzend_extension = /usr/lib/php/20190902/xdebug.so\n[XDebug]\nxdebug.remote_enable = 1\nxdebug.remote_autostart = 1\n' >> /etc/php/7.4/cli/php.ini"

# Export environment variables
ENV DATABASE_USER=root
ENV DATABASE_HOST=127.0.0.1
ENV DATABASE_PASSWORD=

Our gitpod.yml

image:
  file: .gitpod.Dockerfile

# List the start up tasks. You can start them in parallel in multiple terminals. See https://www.gitpod.io/docs/config-start-tasks/
tasks:
    - init: ./setup.sh
    - before: wget https://get.symfony.com/cli/installer -O - | bash  
      command: export PATH="$HOME/.symfony/bin:$PATH" && symfony server:start

In setup.sh we are trying to install our databases:

#!/bin/bash

# Install databases
mysql -e 'CREATE DATABASE app;'
mysql -e 'CREATE DATABASE app_test;'

# Install Composer dependencies
composer install

When the node is trying to install our databases we get an error in the terminal output:

ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

If we are running the above commands after initialization is finished, everything works as it should:

gitpod /workspace/aaas-api $ mysql -e "show databases;"
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| sys                |
+--------------------+

Was there anything changed on GitPod or is there something we did wrong? We're pretty sure it worked not too long ago.

Many thanks for your awesome application by the way. 👍

[jankeromnes]

Hi @christiansiewert!

Thank you for providing such detailed information about your setup, this is very helpful to investigate this bug.

At first glance, I'm slightly suspicious about the tasks in your .gitpod.yml:

# List the start up tasks. You can start them in parallel in multiple terminals. See https://www.gitpod.io/docs/config-start-tasks/
tasks:
    - init: ./setup.sh
    - before: wget https://get.symfony.com/cli/installer -O - | bash  
      command: export PATH="$HOME/.symfony/bin:$PATH" && symfony server:start

It seems that you are "racing" ./setup.sh and all the other commands in two separate terminals (every - under tasks: corresponds to one terminal, and all terminals run in parallel).

Maybe the fix is to just replace the second - with a space (in order to run ./setup.sh sequentially in the order before > init > command), but I'll try to set up a minimal repository inspired by your config files to see if I can reproduce a bug.

[jankeromnes]

Ok, my proof-of-concept repository https://github.com/jankeromnes/gitpod-mysql is now able to reproduce the problem:

gitpod /workspace/gitpod-mysql $  HISTFILE=/workspace/.gitpod/cmd-0 history -r; {
> ./setup.sh
> }
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

This looks like a timing issue where ./setup.sh tries to connect to MySQL before the server is actually ready to service requests.

I've been able to resolve this problem by simply waiting for port 3306 (MySQL) to become active before running ./setup.sh. You can do this with a one-liner:

gp await-port 3306 && ./setup.sh

See also jankeromnes/gitpod-mysql@3754992

[christiansiewert]

Hi @jankeromnes and many thanks for your great investigation. I added gp await-port 3306 and it is working like a charme now so maybe it was just a timing issue.

At first glance, I'm slightly suspicious about the tasks in your .gitpod.yml:

Ok, maybe I misunderstood the documentation at this Point. The idea was to run ./setup.sh only on workspace initialization and the commands after before each time the workspace starts.

I think, I will take a look at the docs again. ;)

[jankeromnes]

Awesome! Thanks for confirming that this resolved the issue. 🙂 🙌

Ok, maybe I misunderstood the documentation at this Point. The idea was to run ./setup.sh only on workspace initialization and the commands after before each time the workspace starts.

Maybe the docs are a bit confusing, but this:

tasks:
    - init: ./setup.sh
    - before: wget https://get.symfony.com/cli/installer -O - | bash  
      command: export PATH="$HOME/.symfony/bin:$PATH" && symfony server:start

will:

• on the first workspace start, run ./setup.sh and wget | bash && symfony server:start in two parallel Terminals
• on subsequent workspace restarts, only run wget | bash && symfony server:start

On the other hand, this:

tasks:
    - init: ./setup.sh
      before: wget https://get.symfony.com/cli/installer -O - | bash  
      command: export PATH="$HOME/.symfony/bin:$PATH" && symfony server:start

will:

• on the first workspace start, run wget | bash && ./setup.sh && symfony server:start (probably what you want)
• on subsequent workspace restarts, run wget | bash && symfony server:start

As you can see, only the first workspace start has different behaviors, because init is only run on first workspace start, but the - determines in which Terminal (or "task group" if you will) it will run.

Hope this clarifies this a tiny bit. 😇

[christiansiewert]

Thank you for your further explanations, @jankeromnes. 👍 I'll try this out in the near future.