[bug] Grist self-hosted is not compatible with SSL

[asitemade4u]

Hi all and happy New Year!! Sorry to begin 2022 with a bug report ;o))

1. I have taken advantage of the holidays to switch all our self-hosted applications to SSL, using Nginx Proxy Manager and the free Cloudflare infrastructure.
2. Everything worked just fine for most of our apps except for Grist.
3. I am using the very last version of Grist Core (continuously updated with Watchtower)
4. When accessing Grist from the newly created https address I arrive an error page:

3. If I click Go to man page, nothing happens
4. If I select Sign in from the upper right menu, I get this error:

And the browser Console log says: Failed to load resource: the server responded with a status of 502 ()

Please help as I would like to allow other employees to access our main worksheets that have all been transferred to Grist.

PS. I can send you logs on your private email address if needed.

[paulfitz]

Hi @asitemade4u, happy new year!

One thing you could try is to set environment variables APP_DOC_URL and APP_HOME_URL to whatever way your site will be accessed by the user. If your site is now at https://example.com, you would set those two variables to that (including the https).

502 is a bad gateway error. Could you check what request the browser was making? It may be in the console just before the error, or failing that in the network tab (once you reload the page).

[asitemade4u]

Hi Paul, and thanks for the prompt response.

1. So I changed the docker-compose.yml file as recommended and it worked: I could connect to the main page (with the documents) and Sign in to my Grist account
2. But, when I try to open a document I get a blank page and an error message:

[asitemade4u]

BUT, as you may know, there is a setting in Nginx Proxy Manager that allows for Web Socket support.
I checked it and I now can access the documents.
Case closed! Thanks!

[asitemade4u]

Even if everything seems fine, I have left the console opened and I see errors each time I open an existing document.
As the log includes private info, i will send it to your private email address.

[paulfitz]

Great, glad it worked and you figured out the websocket setting. Will watch my mailbox for the remaining issue.

[paulfitz]

@asitemade4u I think the error you are seeing is a failure to load browser-check.js, a small utility to check for unsupported browsers and give a little warning in them. I wouldn't expect its absence to cause trouble. I'll look into it, thanks for reporting.

[paulfitz]

@asitemade4u browser-check.js should be loading well in latest docker version. Do you have any remaining problems in using your own domain name with https?

[asitemade4u]

Hi Paul,
No problem except for the remaining (and quite stressful) issue with losing the path to documents at each update (I use Watchtower and you issue a lot of updates).
Actually, the (small) issue has gone worst since I have my own domain and certificate: instead of having to Sign in here is how it goes:

1. I arrive on an error page (Something went wrong)
2. I sign in and arrive on a page where my documents are not displayed (this is the stressful part)
3. I sign in again and now view the documents

[paulfitz]

@asitemade4u I think this is a symptom of losing session information. You can control where session information is stored with the environment variable GRIST_INST_DIR. I think setting it to GRIST_INST_DIR=/persist may work in your case, if /persist is a directory mapped onto the host. With that, after you log in, the log in should survive across docker restarts. You'll see an extra grist-sessions.db file. It is also possible to direct Grist to store session information in an external redis instance by setting REDIS_URL.

[asitemade4u]

OK, will try and let you know

[asitemade4u]

Hi Paul,
It did not work, at least with Cloudflare + Nginx Proxy Manager.
So:

1. I changed the docker-compose.yml file according to your recommendations and added - GRIST_INST_DIR=/home/stsinc/dkr/gst/aut to the environment: section
2. I unlocked the "cloud" icon on Cloudflare's DNS line (I "grey-clouded" it)
3. I emptied the cache of my browser

4. I get a Bad Gateway error

[paulfitz]

@asitemade4u is there a volumes section in your docker-compose.yml for the directory /home/stsinc/dkr/gst/aut? If not, you'll need one so the container has access to it. And make sure to use the mapped directory name in the environment variable. For example, when your docker-compose.yml was as you had it in #90, you'd need GRIST_INST_DIR=/persist.

[paulfitz]

@asitemade4u in latest image GRIST_INST_DIR is now set by default to /persist. this will soon be automatic, but isn't yet.

[paulfitz]

GRIST_INST_DIR is now set by default to /persist in latest docker image, meaning session information will survive container upgrades.

[ovizii]

@asitemade4u is there a volumes section in your docker-compose.yml for the directory /home/stsinc/dkr/gst/aut? If not, you'll need one so the container has access to it. And make sure to use the mapped directory name in the environment variable. For example, when your docker-compose.yml was as you had it in #90, you'd need GRIST_INST_DIR=/persist.

I'm just getting started, trying to self-host grist and am wondering if there is a complete list/documentation of all env variables somewhere?
I saw quite a few mentioned here and there in different issues but no comprehensive list.

[paulfitz]

@ovizii good point. We have an internal list for our hosted service, here's a quick stab at pulling out the relevant parts.

Grist environment variables

Grist is sensitive to several environment variables. Here is a list:

Variable	Purpose
APP_UNTRUSTED_URL	URL to serve plugin content from, or where to expect to find it
GRIST_APP_ROOT	directory containing Grist code, including subdirectories such as sandbox, static, and bower_components.
GRIST_DATA_DIR	directory to search for document list.
GRIST_DOC_BUCKET	s3 bucket where Grist documents are stored, for hosted Grist.
GRIST_DOMAIN	in hosted Grist, Grist is served from subdomains of this domain. Defaults to "getgrist.com".
GRIST_HOST	hostname to use when listening on a port (not respected for electron).
GRIST_INST_DIR	path to Grist instance configuration files, for Grist server.
GRIST_LOGIN_REDIRECT_HOST	host of cognito-based login helper, if applicable (usually login.getgrist.com).
GRIST_MAX_UPLOAD_ATTACHMENT_MB	Max allowed size for attachments (0 or empty for unlimited).
GRIST_MAX_UPLOAD_IMPORT_MB	Max allowed size for imports (except .grist files) (0 or empty for unlimited).
GRIST_SESSION_COOKIE	if set, overrides the name of grist's cookie
GRIST_SESSION_DOMAIN	if set, associates the cookie with the given domain - otherwise defaults to GRIST_DOMAIN
GRIST_SESSION_SECRET	the key used to encode the session
GRIST_TAG	internal grist urls are /v/<GRIST_TAG>/....
GRIST_TESTING_SOCKET	a socket used for out-of-channel communication during tests only.
GRIST_TEST_HTTPS_OFFSET	if set, adds https ports at the specified offset. This is useful in testing.
GRIST_TEST_SSL_CERT	if set, contains filename of SSL certificate.
GRIST_TEST_SSL_KEY	if set, contains filename of SSL private key.
GRIST_TEST_LOGIN	allow fake unauthenticated test logins (suitable for dev environment only).
GRIST_USER_ROOT	an extra path to look for plugins in.
HOME_PORT	port number to listen on for REST API server; if set to "share", add API endpoints to regular grist port.
PORT	port number to listen on for Grist server
REDIS_URL	optional redis server for browser sessions and db query caching
GOOGLE_CLIENT_ID	set to the Google Client Id to be used with Google API client
GOOGLE_CLIENT_SECRET	set to the Google Client Secret to be used with Google API client
GOOGLE_API_KEY	set to the Google API Key to be used with Google API client (accessing public files)
GOOGLE_DRIVE_SCOPE	set to the scope requested for Google Drive integration (defaults to drive.file)

Database variables:

Variable	Purpose
TYPEORM_DATABASE	database filename for sqlite or database name for other db types
TYPEORM_HOST	host for db
TYPEORM_LOGGING	set to 'true' to see all sql queries
TYPEORM_PASSWORD	password to use
TYPEORM_PORT	port number for db if not the default for that db type
TYPEORM_TYPE	set to 'sqlite' or 'postgres'
TYPEORM_USERNAME	username to connect as

Grist hosting:

Variable	Purpose
APP_DOC_URL	doc worker url, set when starting an individual doc worker (other servers will find doc worker urls via redis)
APP_STATIC_URL	url prefix for static resources (all servers need this)
APP_HOME_URL	url prefix for home api (home and doc servers need this)
APP_DOC_PORT	port to be used by an individual doc worker
APP_STATIC_PORT	port to be used by an individual static server
APP_HOME_PORT	port to be used by an individual home server
APP_DOC_START	set to "1" if container should run a doc worker
APP_STATIC_START	set to "1" if container should run a static server
APP_HOME_START	set to "1" if container should run a home server
GRIST_ADAPT_DOMAIN	set to "true" to support multiple base domains (careful, host header should be trustworthy)
GRIST_BACKUP_DELAY_SECS	wait this long after a doc change before making a backup
GRIST_DEFAULT_EMAIL	if set, login as this user if no other credentials presented
GRIST_ID_PREFIX	for subdomains of form o-, expect/produce o-${GRIST_ID_PREFIX}.
GRIST_SINGLE_ORG	set to an org "domain" to pin client to that org
GRIST_ORG_IN_PATH	if true, encode org in path rather than domain
GRIST_HOME_INCLUDE_STATIC	if set, home server also serves static resources
GRIST_ROUTER_URL	optional url for an api that allows servers to be (un)registered with a load balancer
GRIST_MANAGED_WORKERS	if set, Grist can assume that if a url targeted at a doc worker returns a 404, that worker is gone
GRIST_SERVE_SAME_ORIGIN	set to "true" to access home server and doc workers on the same protocol-host-port as the top-level page, same as for custom domains (careful, host header should be trustworthy)
GRIST_SUPPORT_ANON	if set to 'true', show UI for anonymous access (not shown by default)
GRIST_TEST_ROUTER	if set, then the home server will serve a mock version of router api at /test/router
GRIST_THROTTLE_CPU	if set, CPU throttling is enabled
GRIST_EXPERIMENTAL_PLUGINS	enables experimental plugins

Sandbox related variables:

Variable	Purpose
GRIST_SANDBOX_FLAVOR	Can be pynbox, unsandboxed, docker, gvisor, macSandboxExec (note: not all options in grist-core). If set, forces Grist to use the specified kind of sandbox.
GRIST_SANDBOX	A program or image name to run as the sandbox. See NSandbox.ts for nerdy details.
PYTHON_VERSION	Can be 2 or 3. If set, documents without an engine setting are assumed to use the specified version of python. Not all sandboxes support all versions.
PYTHON_VERSION_ON_CREATION	Can be 2 or 3. If set, newly created documents have an engine setting set to python2 or python3. Not all sandboxes support all versions.

[ovizii]

Thank you very much for the effort!

[duythongle]

Got the same issue here. I set up Grist with Cloudflare tunnel as the following docker-compose

version: '3.3'
services:
  grist-core:
    container_name: grist-core
    ports:
      - "127.0.0.1:8484:8484"
    volumes:
      - "~/grist-core:/persist"
    environment:
      - GRIST_SESSION_SECRET="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

    image: gristlabs/grist
    restart: unless-stopped
  tunnel:
    container_name: cloudflared-tunnel-grist-core
    image: cloudflare/cloudflared
    restart: unless-stopped
    network_mode: host
    command: tunnel --no-autoupdate run
    environment:
      - TUNNEL_TOKEN=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Chrome Developer tab shows some incorrect API calls at https://mydomain/o/docs/ which, I guess, caused the issue:

http://mydomain:8484/o/docs/api/session/access/active
http://mydomain:8484/o/docs/api/session/access/all
http://mydomain:8484/o/docs/api/log

The correct root URL should be https://mydomain/

What environment should I set? (@paulfitz )

[paulfitz]

APP_HOME_URL would be the key variable to set @duythongle - see https://support.getgrist.com/self-managed/#how-do-i-run-grist-on-a-server

[duythongle]

@paulfitz
I added APP_HOME_URL="https://mydomain". It throws an error when running

Welcome to Grist.
In quiet mode, see http://localhost:8484 to use.
For full logs, re-run with DEBUG=1
2023-09-03 19:29:58.225 - info: == Grist version is 1.1.3 (commit unknown)
2023-09-03 19:29:58.231 - debug: skipping incomplete language fa (set GRIST_OFFER_ALL_LANGUAGES if you want it)
TypeError [ERR_INVALID_URL]: Invalid URL: "https://mydomain"
    at new NodeError (internal/errors.js:322:7)
    at onParseError (internal/url.js:270:9)
    at new URL (internal/url.js:346:5)
    at new FlexServer (/grist/_build/app/server/lib/FlexServer.js:153:73)
    at main (/grist/_build/app/server/mergedServerMain.js:39:20)
    at main (/grist/_build/stubs/app/server/server.js:124:54) {
  input: '"https://mydomain"',
  code: 'ERR_INVALID_URL'
}

Please note that I run the app on localhost only ports: "127.0.0.1:8484:8484" to hide the origin server from public internet. Cloudflare tunnel helps publish the app to Cloudflare platform with mydomain. The issue can be re-produced easily with my docker-compose above.

[paulfitz]

@duythongle I think somehow the quotes are getting included in your URL, since if I try an invalid URL:

$ docker run -e APP_HOME_URL=zig -it gristlabs/grist
Welcome to Grist.
In quiet mode, see http://localhost:8484 to use.
For full logs, re-run with DEBUG=1
2023-09-03 19:52:40.200 - info: == Grist version is 1.1.3 (commit unknown)
2023-09-03 19:52:40.206 - debug: skipping incomplete language fa (set GRIST_OFFER_ALL_LANGUAGES if you want it)
TypeError [ERR_INVALID_URL]: Invalid URL: zig

There are no quotes in the error message.

And input: '"https://mydomain"' suggests also the quotes are getting included.

[duythongle]

@paulfitz
Removing the quotes to APP_HOME_URL=https://mydomain helps resolve the issue. The app works as expected. Many thanks.