Localhost development?

[mhpaler]

I have a production site up and running and I'm finding it very tricky to try to develop the DoG setup on live servers. Has anyone gotten things setup locally in order to develop/test DoG?

[vikaspotluri123]

Setting up Discourse was kind-of painful for me, but DoG and Ghost should be relatively straightforward!

Discourse
I got Discourse set up using Install Discourse for development using Docker. It was a bit annoying for me because I run docker in a dev VM, so I had to muck with the development configuration to make it respond to requests on my .local domain, and I still have issues with it - I have to replace localhost:3000 with vm.local:4200 a lot.

There might be an easier way to get stuff set up, but this is me and my limited knowledge of Discourse 😄

Ghost
You should be able to ghost install local to get Ghost up-and-running.

DoG
You can follow the installation instructions, but skip setting up systemd, and possibly nginx (depending on how your local environment is configured).

For DoG development, you can clone, yarn install, set up your .env based on the configuration guide, and run yarn dev.

It does take a bit of time to get everything up and running, but hopefully there's not any points where you get stuck. If you do, feel free to post a question so we can figure it out 👍

[mhpaler]

@vikaspotluri123 ok, been giving is a solid "college try" here -- but need to throw in the towel.

Got DoG service running as follows:

For DoG development, you can clone, yarn install, set up your .env based on the configuration guide, and run yarn dev.

Service seems to running fine:

User@swaylock ~/Sites/discourse-on-ghost-dev (master)$ yarn dev                                                                      [2.3.0]
yarn run v1.22.17
$ node scripts/dev.js
Watching for changes...
[2023-08-07 19:34:55] INFO Listening on http://127.0.0.1:3286

Added the hack into {ghost}/current/core/boot.js#bootGhost as you suggested, specifically:

// Step 2 - Start server with minimal app in global maintenance mode
debug('Begin: load server + minimal app');
const rootApp = require('./app')();

const dog = await import('/Users/User/Sites/discourse-on-ghost-dev/dist/targets/dev-proxy.js');
await dog.load('/Users/User/Sites/discourse-on-ghost-dev/dist/', rootApp);

However, ghost would not restart, and provided very little to debug with:

Running in development mode

✔ Checking system Node.js version - found v16.15.0
✔ Checking current folder permissions
✔ Validating config
✔ Checking memory availability
✔ Checking binary dependencies
✖ Starting Ghost: ghost-local
A GhostError occurred.

Message: Ghost process exited with code: 0

Debug Information:
    OS: macOS, v12.6.7
    Node Version: v16.15.0
    Ghost Version: 5.38.0
    Ghost-CLI Version: 1.24.2
    Environment: development
    Command: 'ghost start'

I tried digging in a bit deeper, specifically, running

User@swaylock ~/Sites/discourse-on-ghost-dev (master)$ node dist/targets/node-first-run.js  

and got an error there:

file:///Users/User/Sites/discourse-on-ghost-dev/dist/services/discourse-sync.js:23
    for (const tier of tiers) {
TypeError: tiers is not iterable

Update: I the the above error is related to Ghost running under http vs https on the localhost?

And that's where I decided to throw in the towel. Obviously, with your generous support, we could go back and forth debugging, but I think that's probably not the best use of our time. In the interest in DoG development, perhaps it's best if you write up a tutorial for getting the setup running locally? I realize that may not be feasible given the wide range of dev platforms, but that's my suggestion.

It's all ok though, I have a work around, and that's setting everything up online on dev servers (Digital Ocean Droplets) and testing things in that framework. Not ideal but it's a path forward!!

Will continue to report back here as I make progress on this excellent service.

Thanks again!

[mhpaler]

I should mention that I would be glad to test drive and provide feedback on any instructions you build up.

[vikaspotluri123]

@mpaler I appreciate all your feedback, and your willingness to contribute! I didn't prioritize documenting the development process since there was little benefit to doing so. Now that there's a bigger benefit, I'll work on improving it 😄

To your issues:

• The hack looks good!
• It turns out that dev-proxy has been broken for a while, but I didn't catch it because it's only needed for session mode, and my previous effort was making jwt mode a reality 😬 It should be fixed now.
• A helpful troubleshooting tip - if ghost fails to start like that, you can try checking the ghost logs (ghost log --error), or running Ghost in your terminal (ghost run --development)
• I've improved the error handling of tiers syncing and the first run entrypoint - if you try running it again, you should get a proper error message now.

[mhpaler]

I've improved the error handling of tiers syncing and the first run entrypoint - if you try running it again, you should get a proper error message now.

Did you push some code? I'm not seeing any...

[vikaspotluri123]

Yep, 3 commits: db22216...master

[mhpaler]

@vikaspotluri123 so I've been banging on this for a few hours -- and I'm observing a strange behavior that I've boiled down to the following:

On a DoG installation on Digital Ocean (where Ghost 5.56 and Discourse are running on their own droplets), and DoG is set up as "session" path (running alongside Ghost), redirects work perfectly. eg. Click login on Discourse, get redirected to Ghost, go through login flow, click Magic link, get redirected back to Discourse authenticated. I'm using the /landingpages/not-logged-in.hbs.

But...

On the local installation, with Ghost 5.58 and Discourse running locally, and DoG running on the dev-proxy, when I click on a magic link, for example:

http://localhost:2368/members/?token=sEOXLFKXxome9s__8TxdZ95WQIWXkJU1&action=signin&r=http%3A%2F%2Flocalhost%3A2368%2Fsignin%2F%3Fsso%3Dbm9uY2U9ZTIyMTRhM2QwMjJjMmNiZGUzMWY4YjM3NzhjZTEzNTgmcmV0dXJuX3Nzb191cmw9aHR0cCUzQSUyRiUyRmxvY2FsaG9zdCUzQTMwMDAlMkZzZXNzaW9uJTJGc3NvX2xvZ2lu%26sig%3D0da703547672dbc88fe74d42cff17e1d76ba5c55a103c34cd99fdad261664607

I can see the DoG is receiving the redirect from Ghost (see below; note "signedIn=true")

User@swaylock ~/Sites/discourse-on-ghost-mp (v0.2.2-beta.0)$ yarn dev                                                                         [2.3.0]
yarn run v1.22.17
$ node scripts/dev.js
Watching for changes...
[2023-08-12 00:19:35] INFO Listening on http://127.0.0.1:3286
[2023-08-12 00:19:48] INFO "GET /ghost/api/external_discourse_on_ghost/sso?sso=bm9uY2U9ZTIyMTRhM2QwMjJjMmNiZGUzMWY4YjM3NzhjZTEzNTgmcmV0dXJuX3Nzb191cmw9aHR0cCUzQSUyRiUyRmxvY2FsaG9zdCUzQTMwMDAlMkZzZXNzaW9uJTJGc3NvX2xvZ2lu&sig=0da703547672dbc88fe74d42cff17e1d76ba5c55a103c34cd99fdad261664607" 302 6ms
[2023-08-12 00:20:17] INFO "GET /ghost/api/external_discourse_on_ghost/sso?sso=bm9uY2U9ZTIyMTRhM2QwMjJjMmNiZGUzMWY4YjM3NzhjZTEzNTgmcmV0dXJuX3Nzb191cmw9aHR0cCUzQSUyRiUyRmxvY2FsaG9zdCUzQTMwMDAlMkZzZXNzaW9uJTJGc3NvX2xvZ2lu&sig=0da703547672dbc88fe74d42cff17e1d76ba5c55a103c34cd99fdad261664607&signedIn=true" 302 1ms

However, DoG is not redirecting to Discourse!! It redirects back to the Ghost DOG_SSO_NO_AUTH_REDIRECT page.

other than the DoG dev-proxy, and the slight difference of Ghost versions, I'm almost certain the setups on Digital Ocean and my localhost are identical.

I realize it's pretty hard for you to debug from there -- but is there anything you can suggest as to why this could be happening?

Thanks!

[vikaspotluri123]

There are a couple scenarios I can think of:

1. Portal didn't have time to actually sign the user in, so DoG is correctly tracking the user as not signed in - you can check if this is the case by getting a magic link but removing everything from sig - e.g. http://localhost:2368/members/?token=sEOXLFKXxome9s__8TxdZ95WQIWXkJU1&action=signin&r=http%3A%2F%2Flocalhost%3A2368%2Fsignin%2F%3Fsso%3Dbm9uY2U9ZTIyMTRhM2QwMjJjMmNiZGUzMWY4YjM3NzhjZTEzNTgmcmV0dXJuX3Nzb191cmw9aHR0cCUzQSUyRiUyRmxvY2FsaG9zdCUzQTMwMDAlMkZzZXNzaW9uJTJGc3NvX2xvZ2lu%26, and then visiting the magic link with the sig parameter. Without sig will sign the user into Ghost, but not redirect them to portal. With sig, the user won't be signed in to Ghost because they're already logged in, but it should redirect the user to DoG.
2. There's something wonky going on with cookies. You can check this by looking for the members cookie on a Ghost frontend page, and checking if that cookie is being sent to DoG.

[mhpaler]

@vikaspotluri123 thanks for the tips. After some more banging, I'm making some progress. I think the problem is neither 1 or 2, but rather something related to Discourse running in development mode in Docker and how it handles ports and services, etc. Stuff I'm not that familiar with.

Some things to note before I describe my progress:

1. I restarted Discourse following these instructions, but this time I started it with the -p flag as follows:

// If you wish to globally expose the ports from the container to the network (default off) use
d/boot_dev --init -p

I hadn't done that previously, and it rang a bell cause I was suspecting a problem with ports.

2. Additionally, for DoG .env vars, all urls use 127.0.0.1 rather than localhost, which I was using before.
3. Finally, before each test, I'm flushing all cookies and history from my testing browser.

Anyway, here's what I'm seeing now. Bear with me it's kinda noodly ;)

1. Goto Discourse. Navigate to a deep link for example, a post in the Site Feedback category that comes pre-installed with Discourse. http://localhost:4200/t/about-the-site-feedback-category/1
2. Click "reply", which prompts Discourse to redirect me to the Ghost signin page configured at DOG_SSO_NO_AUTH_REDIRECT
3. Enter a registered user email. Ghost emails me a magic link (I have Ghost configured to use an SMTP server).
4. Copy the link out of my email client and paste it back into my testing browser. here's a look at the link. All looks good!:

http://127.0.0.1:2368/members/?token=RFrig1aBFrc_46Z98WdcnHlFt9eHI3pU&action=signin&r=http%3A%2F%2F127.0.0.1%3A2368%2Fsignin%2F%3Fsso%3Dbm9uY2U9MDJkMTVlZmEzM2RmY2FlNDJhY2NmNjg1NjJhNDg3MGYmcmV0dXJuX3Nzb191cmw9aHR0cCUzQSUyRiUyRmxvY2FsaG9zdCUzQTMwMDAlMkZzZXNzaW9uJTJGc3NvX2xvZ2lu%26sig%3D6b0c0a59d3340b65a64549357fee7dd249c651f50c373b1b0b8800e5c0fa457a

5. Hit enter, and the browser redirects here:

http://localhost:3000/session/sso_login?sso=bm9uY2U9MDJkMTVlZmEzM2RmY2FlNDJhY2NmNjg1NjJhNDg3MGYmZW1haWw9YWRtaW4lNDBzd2F5bG9ja3MuY29tJmV4dGVybmFsX2lkPTRiMzIxMzQ0LWRhMjYtNDgyNS1iM2MyLTAwN2JhY2RkMTVlOCZhdmF0YXJfdXJsPWh0dHBzJTNBJTJGJTJGd3d3LmdyYXZhdGFyLmNvbSUyRmF2YXRhciUyRmM1YWFmNzBlYzAxNjdmNDcxMDhiYWNjMGNlY2E3ZDUwJTNGcyUzRDI1MCUyNnIlM0RnJTI2ZCUzRGlkZW50aWNvbiZuYW1lPUFkbWlu&sig=f75f804b787781b42847ed434002343d08f49c2af3a6eb6502c1cefaa7dfcec7

And I get a proper Discourse page with an error as shown here:

6. Here's where it gets interesting! If I quickly change the url base in the browser from http://localhost:3000/ to http://127.0.0.1:4200/ and hit enter I get redirected to http://127.0.0.1:4200/t/about-the-site-feedback-category/1

What's more, this only works if I change it to http://127.0.0.1:4200, not http://localhost:4200.

One thing I'm noting is that original redirect when you click on the magic link (http://localhost:3000/session/sso_login?[...]) has localhost in it. Similarly, Discourse system generated emails, for example /u/admin-login generate URLS that have http://localhost as their root. So it seems that Discourse thinks of itself as http://localhost and not http://127.0.0.1??

Anyhow, that's where I'm at. So I think we're really close! What do you think is happening here?

[vikaspotluri123]

Gaaah, that's so frustrating! What's even weirder is Discousre seems to have hardcoded support for localhost: https://github.com/discourse/discourse/blob/91e5fa506d68ccf8f5e944431b505b0fe562a632/config/environments/development.rb#L101

That being said, my setup is about the same as yours now - I manually change the redirect URL to http://vm.local:4200 since the setup was "good enough" to test and I had spent way too long trying to get all the docker stuff set up.

The --init -p requirement is very interesting - I was expecting that to only be required when Docker isn't running on the same machine as your dev environment