hasura/graphql-engine:v1.3.3.cli-migrations-v2 crashes on heroku

[schwamic]

Hi,
after a few months hasura stopped working on heroku, even though there were no changes made to the code.

Error-Message:

{"timestamp":"2021-05-14T08:42:38.000+0000","level":"info","type":"startup","detail":{"kind":"installing cli-ext plugin","info":""}}
FATA[0000] preparing execution context failed: setting up global config failed: cannot get home directory: exec: "getent": executable file not found in $PATH

I deployed hasura on a heroku-container.
Environment varialbes are set.
I use a 7$ Dyno, Western Europe.

Dockerimage:

FROM hasura/graphql-engine:v1.3.3.cli-migrations-v2

COPY ./hasura/migrations /hasura-migrations
COPY ./hasura/metadata /hasura-metadata

CMD graphql-engine \
    --database-url $DATABASE_URL \
    serve \
    --server-port $PORT

heroku.yml

build:
  docker:
    web: Dockerfile

Does anyone have a similar problem or has an idea what the problem might be?

[warrenshen]

I'm running into this exact same issue, as described above (hasura deployed on Heroku).

Server logs:

2021-05-14T16:21:35.136967+00:00 app[web.1]: time="2021-05-14T16:21:35Z" level=fatal msg="preparing execution context failed: setting up global config failed: cannot get home directory: exec: \"getent\": executable file not found in $PATH"

The issue is a result of hasura-cli migrate apply, which is run in a docker-entrypoint.sh file.

[jesus-itravel]

Same problem here (and also 3 days ago). After creation of a new app on heroku (no other change needed) everything worked (just needed to point to the new hasura app where required).

[scriptonist]

@schwamic I just tried to reproduce this on free tier with the given docker file, but seems working for me.

The CLI uses https://github.com/mitchellh/go-homedir to determine the "home directory". This package seem to look first at $HOME environment variable if it's set it'll use that.

ref: https://github.com/mitchellh/go-homedir/blob/af06845cf3004701891bf4fdb884bfe4920b3727/homedir.go#L97

If $HOME is not set, it'll go ahead and see if it can populate the value using getent command

ref: https://github.com/mitchellh/go-homedir/blob/af06845cf3004701891bf4fdb884bfe4920b3727/homedir.go#L114

So, maybe $HOME is not getting set in the container and given that this used to work correctly before few months, maybe some changes on heroku side?

[schwamic]

hi @scriptonist heroku must have changed something, but I don't know what... after running hasura without errors, the image stopped working magically three days ago.

It's very hard to debug the hasura-image on heroku, because I can't get on the vm via ssh (hasura/issue). So it would be a helpful feature, if the hasura-image could be extended so devs are able to ssh to their vm on heroku for debugging-purpose.

1. Did you use hasura/graphql-engine:v1.3.3.cli-migrations-v2 including metadata and migrationfiles? Locally everything is working fine also, chrashes only on heroku...
2. Do you know how to set $HOME/$PATH in heroku?

[scriptonist]

@schwamic It seems like the steps tot enable SSH is mentioned in https://devcenter.heroku.com/articles/exec#using-with-docker

Did you use hasura/graphql-engine:v1.3.3.cli-migrations-v2 including metadata and migrationfiles? Locally everything is working fine also, chrashes only on heroku...

Yes, exactly same as your Dockefile

Do you know how to set $HOME/$PATH in heroku?

Maybe add a line line like CMD ["echo", "$HOME", othercommands...] to your Dockefile ?

[dimitrinicolas]

I tried it on my side, on a $7 Heroku dyno that has the same issue with cli-migration-v2, $JOME is set to "/root" for me.

[schwamic]

@scriptonist if I run CMD ["echo", "$HOME"] I get echo \. @dimitrinicolas could you share your Dockerimage and heroku.yml? And how do you get this specific logs?

I always get this logs:

2021-05-18T16:55:04.044120+00:00 heroku[web.1]: State changed from starting to crashed
2021-05-18T16:55:08.094658+00:00 app[api]: Deploy c8148aae by user xxx
2021-05-18T16:55:08.094658+00:00 app[api]: Release v163 created by xxx
2021-05-18T16:55:08.000000+00:00 app[api]: Build succeeded

2021-05-18T16:55:09.423561+00:00 heroku[web.1]: State changed from crashed to starting
2021-05-18T16:55:10.989599+00:00 heroku[web.1]: Starting process with command `/bin/sh -c graphql-engine\ \ \ \ \ --database-url\ \postgres://xxx.eu-west-1.compute.amazonaws.com:5432/xxx\ \ \ \ \ serve\ \ \ \ \ --server-port\ \18459`
2021-05-18T16:55:12.934911+00:00 heroku[web.1]: Process exited with status 1
2021-05-18T16:55:12.983734+00:00 heroku[web.1]: State changed from starting to crashed
2021-05-18T16:55:12.677372+00:00 app[web.1]: {"timestamp":"2021-05-18T16:55:12.000+0000","level":"info","type":"startup","detail":{"kind":"installing cli-ext plugin","info":""}}
2021-05-18T16:55:12.893271+00:00 app[web.1]: time="2021-05-18T16:55:12Z" level=fatal msg="preparing execution context failed: setting up global config failed: cannot get home directory: exec: \"getent\": executable file not found in $PATH"

[dimitrinicolas]

@schwamic Here is my Dockerfile:

FROM hasura/graphql-engine:v1.3.3.cli-migrations-v2
# FROM hasura/graphql-engine:v1.3.3

COPY ./migrations /hasura-migrations
COPY ./metadata /hasura-metadata

ENV HASURA_GRAPHQL_ENABLE_CONSOLE=true

ENV HASURA_GRAPHQL_ENABLE_TELEMETRY=false

ENV HASURA_GRAPHQL_ENABLED_LOG_TYPES=startup,http-log,query-log,websocket-log,webhook-log

ENV HASURA_GRAPHQL_MIGRATIONS_DATABASE_ENV_VAR=DATABASE_URL

RUN echo $HOME

CMD graphql-engine \
  --database-url $DATABASE_URL \
  serve \
  --server-port $PORT

Myheroku.yml:

build:
  docker:
    web: Dockerfile

I'm seeing the RUN echo $HOME result from the "build log" in Heroku dashboard:

[schwamic]

@dimitrinicolas thank you! @scriptonist after checking on the right place for the logs in heroku (build logs) I also can confirm that the $HOME is set to /root. So could it be that there is an issue in combination with go-homedir?

[warrenshen]

For those interested in a workaround, creating a new app on Heroku (what @jesus-itravel mentioned) worked for me.

[schwamic]

@warrenshen thank you for this information! For me, unfortunately, this is not so easy to do. but then I think it is an issue on heroku side... I hope the heroku support will get back to me soon and I'll share my insights here :-)

[schwamic]

@scriptonist do you have some answers to this response from heroku?

There are a few reasons why I don't believe this is a Heroku problem:

1. There is only one filesystem in a dyno, a dyno could not possibly still be running without it and an environment.
2. On a regular (non-Docker) Heroku built dyno, $HOME is set to /app. If the Docker env overrides this, it is then up to the app to manage details that would otherwise depend on it.
3. The use of Docker removes the use of many Heroku container assumptions, though still one filesystem, it still requires an environment, etc.

Though, reading through that GitHub thread I'm interested in the home directory being /root, especially given that the user in the container does not run as a root user ( https://devcenter.heroku.com/articles/container-registry-and-runtime#run-the-image-as-a-non-root-user ).

I'm unable to provide a specific reason for this error, as app specific diagnosis is outside of the scope of Heroku Support. I would be able to comment further if it were a platform issue, but I don't believe that to be the case. If you have proof that it may be, please provide more details about it.

It is strange to me that creating a new app is sufficient to work around this, but the details of the cause of this bug, over a period of time it takes to occur, is it up to the developer to determine.

This is how the root looks like: ($HOME = /root and user = root)

drwxr-xr-x    1 root     root            69 May 19 06:24 .
drwxr-xr-x    1 root     root            69 May 19 06:24 ..
-rwxr-xr-x    1 root     root             0 May 19 06:24 .dockerenv
drwxr-xr-x    2 root     root         12288 Nov 24 10:18 bin
drwxr-xr-x    5 root     root           340 May 19 06:24 dev
drwxr-xr-x    1 root     root            66 May 19 06:24 etc
drwxr-xr-x    2 root     root           216 May 19 06:24 hasura-metadata
drwxr-xr-x   18 root     root          4096 May 19 06:24 hasura-migrations
drwxr-xr-x    2 root     root             6 Jul 24  2019 home
drwxr-xr-x    3 root     root           306 Nov 24 10:15 lib
lrwxrwxrwx    1 root     root             3 Jul 24  2019 lib64 -> lib
drwxr-xr-x    4 root     root            71 Nov 24 10:18 opt
dr-xr-xr-x  172 root     root             0 May 19 06:24 proc
drwx------    2 root     root             6 Jul 24  2019 root
drwxr-xr-x    3 root     root            28 Nov 24 10:15 srv
dr-xr-xr-x   13 root     root             0 May 19 06:24 sys
drwxrwxrwt    2 root     root             6 Jul 24  2019 tmp
drwxr-xr-x    5 root     root            42 Nov 24 10:17 usr

If I build the hasura-image, there are also a lot existing layers. I thought this could also be interesting to know, because some people solved that issue by creating a new app on heroku...

5f9f8540460b: Preparing
db292dac57ac: Preparing
be353d6bb0bc: Preparing
af920a143447: Preparing
374160c337ab: Preparing
d7f01a915407: Preparing
d13220829ac5: Preparing
2165e75931bb: Preparing
dc9c136c5a2f: Preparing
cb11bf7a262c: Preparing
d7f01a915407: Waiting
d13220829ac5: Waiting
2165e75931bb: Waiting
dc9c136c5a2f: Waiting
cb11bf7a262c: Waiting
af920a143447: Layer already exists
374160c337ab: Layer already exists
d7f01a915407: Layer already exists
d13220829ac5: Layer already exists
2165e75931bb: Layer already exists
dc9c136c5a2f: Layer already exists
cb11bf7a262c: Layer already exists
5f9f8540460b: Pushed
db292dac57ac: Pushed
be353d6bb0bc: Pushed

[scriptonist]

@schwamic thank you for getting back with the information. I'll spend some time on this and get back to you with my results/findings.

[scriptonist]

@schwamic from the docker file RUN echo $HOME runs during an image build right? IIUC It has to be in CMD block so that it runs inside the container right?

[schwamic]

@scriptonist I could fix it :-) this dockerfile is working on heroku:

FROM hasura/graphql-engine:v1.3.3.cli-migrations-v2

# Add migrations and metadata
COPY ./migrations /hasura-migrations
COPY ./metadata /hasura-metadata
WORKDIR /

# Run the image as a non-root user
RUN adduser -D appuser
USER appuser

# Run the app. CMD is required to run on heroku
CMD graphql-engine \
    --database-url $DATABASE_URL \
    serve \
    --server-port $PORT

[dimitrinicolas]

@schwamic Amazing, it works for me too. Thank you so much!

[rodrigoreis22]

@schwamic Nice, the WORKDIR command did it for me. Thank you!

[namoscato]

Heroku dropped a related changelog item today – months after we also experienced this issue (which was resolved by the WORKDIR / solution). We worked with Heroku Support on this a bit to try and identify a root cause without luck, but this might be it?