Self-managed Grist onboarding is confusing

[edward-shen]

Hey folks.

I'd like to give feedback on the onboarding experience for a self-managed instance. Specifically, I think the documentation needs significant improvement for anyone to have a reasonable chance at onboarding.

I just got to a point where logging in doesn't give any error messages. I have not interacted with any of grist yet, nor set up sandboxing. I have instead set up requiring login and disabling anon flows to have a public instance.

It took me 4 hours to get to this point, judging by stat.

As of right now, the guide results in generating the following environment variables:

      - APP_HOME_URL=https://grist.<DOMAIN>
      - GRIST_SESSION_SECRET=<SECRET>
      - GRIST_SINGLE_ORG=<any-org-name>
      # a bunch of SAML auth, which works for now

As well as to mount the /persist directory in the docker container and to forward some ports.

This results in a non-functioning instance if you have a reverse proxy that does not set the Host header to the correct domain. Specifically, this results in #228, where the frontend seems to ignore APP_HOME_URL.

---

Feedback: It's not sufficient to just say

You will need to place a “reverse proxy” in front of Grist to handle “ssl termination” (decrypting encypted traffic) using a certificate that establishes ownership of the site. If you don’t know what this means, you could try using the Grist Omnibus which packages Grist with a reverse proxy that will use Let’s Encrypt to get a certificate for you automatically.

on the initialization guide when your business logic extracts and utilizes specific headers that a reverse proxy (e.g. nginx) is not guaranteed to provide, especially when it's heavily implied that just setting APP_HOME_URL is all you need to do.

(And if you're wondering which header is omitted, see #228; I've commented there with more details.)

I assume there's also some websocket configuration I was able to avoid because I just copied my nginx config from another virtual server's location block that uses websockets.

---

This led me to trying out a bunch of environment variables, reading source code, and trying to figure out why APP_HOME_URL wasn't being respected. Turns out, ignoring APP_HOME_URL is intentional as there's a function that overwrites it if GRIST_SERVE_SAME_ORIGIN is set, which is set to true for the Dockerfile.

---

Feedback: Log more. DEBUG=1 gives near-zero meaningful information when we're trying to debug how things are constructed. I often felt it was probably easier for me to just pull source and run it locally with locally added log statements.

Feedback: Log a LOT more. I really would have loved a trace level to figure out where the client-side urls were being overwritten.

Feedback: Tell us what env variables are set during initialization. This lets the user verify that the env vars have been properly set.

---

But once this was fixed, I still ran into problems -- going to https://grist.<DOMAIN> and logging in lead me to an auth error where "I don't have access to this organization's documents" and I'll need to contact an administrator:

---

Feedback: What organization? What administrator? I am the administrator!? Where am I suppose to set up a organization? Is there an administrator account?

---

Click on the Grist icon then leads you to a blank page. No, really:

The server side reveals no error message, so it must have served everything fine. It's only in the console logs that reveals the actual issue:

Error: Cannot figure out what organization the URL is for.

---

Feedback: There needs to be some error handling here. There's no way for a user to figure out what the issue is here, especially if they were just given a bad link.

---

Figuring why this was broken required me to look at #212. It looks like the default organization is actually docs, and not some random org I could set. It also looks like I needed to set GIRST_ORG_IN_PATH to have the client figure out what I could set:

      - GRIST_SINGLE_ORG=docs
      - GRIST_ORG_IN_PATH=true

I'm pretty sure this is unintentional but this was the first thing that worked for me.

---

Feedback: More docs on figuring out how to set a single org instance up is incredibly helpful.

---

Unsetting GRIST_SINGLE_ORG and just keeping GIRST_ORG_IN_PATH enabled the "Create a new Team site" button, but just redirected me to the pricing page instead. I'm not sure if this is intentional or not, since I feel like grist-core should support team pages.

Finally, once that got working, I nuked the instance and set these variables, as this is a publicly accessible instance and I want it to be gated on my SAML provider:

      - GRIST_FORCE_LOGIN=true
      - GRIST_ANON_PLAYGROUND=false

I started things up again and I'm finally able to get to a starter page. I take one last look at the logs and I see that my user account is Id 5:

ie.sh/
grist_1  | 2023-11-09 08:27:09.083 grist.<DOMAIN> POST /saml/assert 302 27.195 ms - 90
grist_1  | 2023-11-09 08:27:09.184 - debug: Auth[GET]: grist.<DOMAIN> / customHostSession=, method=GET, host=grist.<DOMAIN>, path=/, org=docs, email=<USER>@<DOMAIN>, userId=5, altSessionId=<SNIP>

Wait, shouldn't I be 1? Looking at the logs, I see this line:

grist_1  | 2023-11-09 08:27:05.051 - debug: Auth[GET]: grist.<DOMAIN> / customHostSession=, method=GET, host=grist.<DOMAIN> path=/, org=docs, email=anon@getgrist.com, userId=1, altSessionId=<SNIP>

Why is the first user anon@getgrist.com, even when I explicitly disabled anon login? Does that mean there's still an anon account that could still perform actions? Who are userIds 2-4?

There's not much constructive feedback regarding this, though I really do wish that there was a way to guarantee that the anon user doesn't exist.

---

At this point in time, I'm still trying to figure out a couple of things:

• The extracted name is just the local-part of my email, so I need to figure out why my Authentik isn't providing the name_id correctly.
• I have no idea how to administer and create a team space -- I only have access to the personal space (prefixed with @)

But hopefully this is valuable feedback!

[paulfitz]

Hi @edward-shen, thanks for synthesizing all that feedback! This is a very valuable thread.

On the narrow question of the low-number userIds. Grist allocates a set of special user ids unconditionally, before doing anything else:

grist-core/app/gen-server/lib/HomeDBManager.ts

Lines 389 to 407 in 23782fd

	public async initializeSpecialIds(options?: {
	skipWorkspaces?: boolean // if set, skip setting example workspace.
	}): Promise<void> {
	await this._getSpecialUserId({
	email: ANONYMOUS_USER_EMAIL,
	name: "Anonymous"
	});
	await this._getSpecialUserId({
	email: PREVIEWER_EMAIL,
	name: "Preview"
	});
	await this._getSpecialUserId({
	email: EVERYONE_EMAIL,
	name: "Everyone"
	});
	await this._getSpecialUserId({
	email: SUPPORT_EMAIL,
	name: "Support"
	});

Whether a user can log in with a specific user id is a separate question.

Most of our effort and attention has so far been devoted to the SaaS we run, which uses one a particular configuration of Grist. We've tweaked things bit by bit to work for others in other situations. It would be lovely to catch our breath and rationalize things. Your write-up here is a great entry-point for that. Thank you!

[edward-shen]

Hi @paulfitz!

I'd like to apologize if any of the feedback is a little sharp -- I was a little frustrated trying to set it all up but from personal experience I know how valuable fresh, new user experience feedback can be so I chose to write feedback as soon as possible. I've attempted to make them constructive and actionable as possible.

To be clear, I do recognize and appreciate the work put into self-hosting capabilities, especially since it can be often difficult to prioritize over business needs.

The objective of this write-up was to start internal conversations. I'm not expecting much beyond that, so please feel free to close this issue as you see fit.

---

As for the user ID feedback, I don't mind that I'm not the first user. Rather, it's that there exists the possibility for malicious actors to potentially abuse special default accounts (e.g. anon; previewer; etc), especially for self-hosted but not properly secured instances. It might not be too hard to imagine someone abusing the anonymous account to either spam or collect information from poorly secured documents on public exposed instances.

Think self-managed enterprise instances where every user that should have permission is given an account, and those without permission can't even access the app (e.g. they are not given a seat license). In that case, none of the special accounts might be necessary: anonymous users and previewer users should already have a seat license to use; everyone should be disabled because it doesn't scale well; and IT might have their own flow that's not email.

This is all hypothetical, of course, but I hope that this offers a reasonable scenario to why these users should be optional (and preferably opt-in).

[paulfitz]

Don't apologize @edward-shen, your constructive objective was clear and I was in fact impressed by your restraint :-). I imagine most people who hit even a few of these problems just move on and we never hear about it.

I'm not getting into responding bullet by bullet to what you've noted - not out of dismissal, but the contrary; it is clear that there is a lot to do. We will be having those conversations.

[cgalo5758]

Thank you @edward-shen for this! I was encountering all the same errors doing this and spent a lot more than 4 hours on this haha

Another piece of feedback that I'd like to add on top of: "Define what is an organization, or what is a team anywhere on the documentation": Please document how to map the SAML roles/attributes to teams or single org when you do define what team, org , admin are

[edward-shen]

@cgalo5758 Were you able to get different orgs to work? By that question it sounds like you were able to manage to achieve that. If so, could you post how you did that?

[cgalo5758]

@edward-shen No, but I thought that maybe roles/attributes had to do with it and that it was simply not documented. I had to add a "User Attribute Mapper For NameID" to make grist fetch the user's email, first name, and last name correctly. For documentation's sake, it is "urn:oasis:names:tc:SAML:1.1:nameid-format:unspecified"... nothing else will work.

[mxcd]

Hi @edward-shen
Thank you so much for your detailed description here. I am at the very same point, where I am trying to get a team page working so I can share documents with my team members by default.

1. Did you manage to surpass the "Access Denied" page when using a GRIST_SINGLE_ORG ?

Maybe @paulfitz
2. What is the difference between the GRIST_SINGLE_ORG: docs and an arbitrary GRIST_SINGLE_ORG: my-org?
3. Using SAML, how am I able to log into the "admin" account of the instance? It would be fine if I could map the SAML user to his organization manually, but it is nowhere written on how to obtain admin credentials, nor how to navigate to a username:password login with SAML being enabled.

Thanks for putting in the effort of making this a free, self-hosted solution along the way

[edward-shen]

I have not been able to figure out how to get team pages working (I run a single person instance)--setting the single org name to docs gets you personal pages though -- not sure if it works for multiple people.

[mxcd]

@edward-shen Thanks for coming back to me.
Setting GRIST_SINGLE_ORG: docs does not enable "document sharing" between different users from what I have tested.
Any advice @paulfitz ?

[paulfitz]

@mxcd a GRIST_SINGLE_ORG setting of docs I believe will limit the Grist installation to distinct personal sites for each user (like what you would see on https://docs.getgrist.com). Using any other setting should give you a "team site" with team sharing options. You mention getting an access denied in that situation though? If you have set GRIST_DEFAULT_EMAIL to an email, then the user with that email will be the initial owner of the team site. If you already ran Grist for a particular GRIST_SINGLE_ORG setting without GRIST_DEFAULT_EMAIL, then the team site will have already been created. You could try a different setting to make a new team site with the right owner. Not sure if that is your problem, I'm just guessing.

I see that fixing the ability to create team sites in grist-core has been buried in our backlog for over a year now. @dsagal should I just post this as a help-wanted issue? It is a fairly simple change, and it is a recurring source of confusion.

[illode]

If you have set GRIST_DEFAULT_EMAIL to an email, then the user with that email will be the initial owner of the team site.

The README also says

if set, login as this user if no other credentials presented
which makes it sounds like it will have the lowest level of authorization. But when GRIST_SINGLE_ORG is set to anything except docs, they're the admin.

---

I fiddled with various configs for a bit, and this is what I found. The behavior seems a bit buggy right now. GRIST_ORG_IN_PATH=false seems to be straight up broken if the team/org name is unset. These were all done stateless, so grist config didn't persist when I restarted it with new env vars. Redis, minio + snapshots, and OIDC were also configured.

With GRIST_SINGLE_ORG unset + GRIST_ORG_IN_PATH=false:

• Navigating to the APP_HOME_URL leads to a "Page not Found" error
• Clicking "Go to main page" just leads back to this page, since it's the main page.
• The "@<user>" org/team in the switch sites section is a link to docs.yourdomain.com, e.g. docs.example.com, which isn't even a real subdomain.
• The "Create new team site" button is there despite GRIST_ORG_IN_PATH=false, it just does literally nothing.
• Manually navigating to grist.yourdomain.com/o/docs just gives a JSON error response: {"error":"Wrong org for this domain: 'docs' does not match 'grist'"}

With GRIST_SINGLE_ORG=docs (or unset) + GRIST_ORG_IN_PATH=true:

• All users get private teams/orgs ("@<user>")
• The anonymous user can be used to create ephemeral documents.
• The multi-user button is there. It doesn't actually work for me, but I suspect that's because of SSO/OIDC.
• The "Create new team site" button leads to the pricing page

With GRIST_SINGLE_ORG=docs + GRIST_ORG_IN_PATH=false:

• Same as above, except there's no "Create new team site" OR multi-user button.

With GRIST_SINGLE_ORG=test123 (anything but docs) + GRIST_ORG_IN_PATH=* (doesn't matter):

• GRIST_FORCE_LOGIN seems to be implicitly enabled, so the anonymous user is useless. APP_HOME_URL redirects straight to SSO login.
• Nobody except for the GRIST_DEFAULT_EMAIL user can access anything. All other users have to be manually added to the team/org by GRIST_DEFAULT_EMAIL, who is effectively the admin user.
• New teams/orgs can't be created, which makes sense given that it's set to single team/org mode
• The multi-user button is gone.
• There's a "Manage Team" button
• Since every user is in the same team, everyone can view every document, which IMO makes this setup strictly worse than just using private teams/orgs.

[vviers]

Happy self-hoster for more than a year here 👋🏼
Fwiw, you can find my team's self-hosting config here (we use K8s but hopefully some of it is useful to you — I know it has helped other self-hosters in the past)

[mxcd]

Thanks to all of your support, I got it working. For reference, here are my ENV vars:

    TZ: Europe/Berlin
    GRIST_SUPPORT_ANON: false
    GRIST_FORCE_LOGIN: true
    COOKIE_MAX_AGE: 86400000
    GRIST_ORG_IN_PATH: true
    GRIST_HIDE_UI_ELEMENTS: billing,sendToDrive # copied from @vviers 
    GRIST_SESSION_SECRET: <SOME_SECRET>
    GRIST_SINGLE_ORG: <my-team-name>
    GRIST_DEFAULT_EMAIL: admin@my-org.com
    APP_HOME_URL: https://grist.my-org.com
    APP_DOC_URL: https://grist.my-org.com
    GRIST_FORWARD_AUTH_HEADER: X-Forwarded-User
    GRIST_FORWARD_AUTH_LOGOUT_PATH: /_oauth/logout
    LOGOUT_REDIRECT: https://grist.my-org.com/signed-out
    # GRIST_SAML_IDP_UNENCRYPTED: true # I needed this to use Keycloak
    GRIST_SAML_SP_HOST: https://grist.my-org.com
    GRIST_SAML_IDP_LOGIN: https://kc.my-org.com/realms/master/protocol/saml
    GRIST_SAML_IDP_LOGOUT: https://kc.my-org.com/realms/master/protocol/saml
    GRIST_SAML_SP_KEY: /saml/sp.key
    GRIST_SAML_SP_CERT: /saml/sp.crt
    GRIST_SAML_IDP_CERTS: /saml/idp.crt

This gives me a single Team page where the contents are shared with all members.
Setting GRIST_DEFAULT_EMAIL: admin@my-org.com makes the user with the defined email the first Owner of the team. New users can be added by using the Manage Team function and stating the email (even if the new user has not yet logged in).

Thanks to everyone for the support ❤️

[tegs]

Hi,

Has anyone been able to successfully setup multi team site in subdomain?

I am able to setup multi team sites with GRIST_ORG_IN_PATH=true and GRIST_SINGLE_ORG unset, but had issues when it comes to duplicating documents.

Tried setting GRIST_SINGLE_ORG to some value and the ORG is noticeably gone from path and duplicate works perfectly as suggested here #185. Reading through the comments made me think that duplication in multi team site as subdomain might work, instead of in path.

So tried setting the server to support subdomain by configuring the web server, local name server and also the Grist environment variable by setting GRIST_ORG_IN_PATH=false and GRIST_SINGLE_ORG unset. However, all links generated always have ORG in path in the form of <DOMAIN.TLD>/o/<ORG> and none appearing as <ORG>.<DOMAIN.TLD>. Are there other environment variables or settings that need to be setup to allow multi team site in subdomain?

[paulfitz]

Multi team site in subdomain is what we use for Grist Lab's SaaS. Looking through our settings, I see GRIST_SESSION_DOMAIN set to .getgrist.com. It is important to have a shared cookie across the installation. The APP_HOME_URL for us is api.getgrist.com - something neutral that can be used for API calls from any site. There aren't other settings that leap to my eye. I don't know of anyone besides us who uses this configuration though, so it may be even rougher to get going than the other options.

It would be great if someone could fix the underlying document duplication issue, it may well be some small URL confusion when someone finally digs into it.

[ecker00]

Think a docker compose example with good defaults and explanations of the various environment variables would go a long way.

[paulfitz]

Update: there is now a /boot page where we are starting to accumulate diagnostics for common problems. The /boot page is somewhat more robust to configuration problems than the rest of the app. Not saying this is a solution to the problems listed in this thread, just a place to chip away at some of them.

[vviers]

Adding to the list : #948 cc @jgranduel

[almereyda]

Multi-team sites on subdomains would be a great addition to self-hosted deployments. Looking forward to sorting that one out.

Related, as it has come up above:

• Official docker-compose file #571

[neuhaus]

Thanks @edward-shen for giving the clues regarding GRIST_SINGLE_ORG=docs and GRIST_ORG_IN_PATH=true, without these options i was getting a main page not found.