Proposal: expose Ghost as reusable express middleware

[nicoburns]

It would be cool if Ghost could be installed as an npm module, and then used as middleware in any express app. So you could do something like:

var express = require("express");
var ghost = require("ghost");

var app = express();

app.get('/', function(req, res) {
  res.send('hello world');
});

var blog = ghost({
    contentDirectory: "./ghost-blog-content"
});

app.use(blog.middleware({
    urlRoot: "/blog"
}));

app.listen(3000);

Notably both the content directory, and the url it is served from are configurable here. It could also be interesting to experiment with allowing the front-end, admin interface and public API to all be served from separate custom locations.

This would allow all sorts of possibilities such as having multiple Ghost blogs in the same server, and embedding ghost as part of a larger CMS product... Obviously this would need to be in addition to it working standalone, but I don't think that would be too difficult to manage (the standalone version could just be a thin wrapper around the middleware version).

Thoughts?

[javorszky]

That is way more advanced than ... although good idea. Why is your avatar a Novell Netware logo, by the way? :)

[ErisDS]

We do intend to refactor to make this possible in the future 👍

[gotdibbs]

Just for reference sake, this came in as a request on the forum today as well: https://en.ghost.org/forum/installation/91-can-i-use-ghost-as-a-module-in-an-existing-express-js-application

[cianclarke]

+1

[rileyjshaw]

+1

[aconran]

+1

[izelnakri]

+1

[ErisDS]

Adding +1 to an open and scheduled issue achieves nothing but sending everyone watching this repo an email. It is very counter productive. If you want the feature, then by all means contribute the code. We understand that this is a desired feature, if we didn't this wouldn't be open & scheduled.

[izelnakri]

I don't think it is counter productive. Contributing to the project means I have to read the source code. This issue is open for 6 months. Achieving this functionality would have the biggest impact for Ghost and its community rather than releasing v0.4.(x+1), that looks more than just an opinion. Lastly, thanks for all the Ghost commits you've made so far. It's the simplest "out-of-the-box" solution to blog I've ever used.

[ErisDS]

Achieving this functionality would have the biggest impact for Ghost and its community rather than releasing v0.4.(x+1), that looks more than just an opinion.

How do you figure this - what impact do you think it will have exactly? Quite clearly, none of the people who want this particular feature are willing to contribute to the project.

[halfdan]

@izelnakri It's also notable that you didn't give any argument why a +1 would be productive. You just said that it's too much work for you to contribute to the project. The +1 adds no value to the discussion/issue. We know that people want this feature and it's scheduled for 0.6. 0.5 has a different focus.

[cianclarke]

Guess code speaks louder than +1's, huh.
https://github.com/TryGhost/Ghost/blob/master/core/index.js#L14
https://github.com/TryGhost/Ghost/blob/master/core/server/index.js#L279
From looking here, it's already possible to pass an existing instance of express when init'ing ghost. The difficulty is it's then binding routes that would be damaging to any existing express application, right?
https://github.com/TryGhost/Ghost/blob/master/core/server/routes/frontend.js#L12
So, I guess the approach would need to be have ghost export something that allows users to mount everything Ghost under a certain route? Something like

app.use('/blog', require('ghost'));

No mean feat I'm sure. I've used instances of HTTPServer that connect middleware can expose in the past to hand off routes like this with much success, but doesn't seem appropriate here. I'm struggling to see what this export would look like? Can an one express instance be bound to another?

It's probably worth figuring how this should work before a member of your community spends time trying to hack something together that doesn't work for you guys..

[hswolff]

I've long wanted this implemented as well but it's easier said then done.

I'm struggling to see what this export would look like?

This is the same road block I encounter when trying to conceptualize a solution. The solution in theory is exactly as you say:

So, I guess the approach would need to be have ghost export something that allows users to mount everything Ghost under a certain route

But again, as you said, harder done than desired :).

I think the light to re-visit this is getting near as there's been a lot of refactoring work done that has made the path to a solution easier to find. However we're not there yet.

Hang tight, it'll find its way into the project eventually! And if you can't wait please do hop into IRC. I'd be overjoyed to discuss some strategies on how to tackle this issue.

[hswolff]

Just saw that Express 4.0 has some new features that may make this also easier to accomplish. Check out this explanation of the new express router. Exciting!

[ErisDS]

This looks pretty promising!

[dougwilson]

Hi! Sometimes I aimlessly browse around GitHub. I just happened to run across this issue. Having Ghost available for mounting in express I'm sure would make a lot of people happy. I have never used Ghost myself, but would be happy to answer any questions regarding what would be the best way to approach this issue :)

[ErisDS]

@dougwilson Making Ghost work as express middleware is something we really want to do. Although it wouldn't provide anything for the majority of our user base, I think it would help us be more appealing to other node developers and perhaps even drive contributions.

The main problem we have is the basic structure of the app - an awful lot happens on boot, including starting the server. Some of this needs breaking down into smaller pieces. Ghost does already support being mounted on subdirectory, so I think that's one of the big pieces - being able to create a node app, and then use Ghost as middleware to serve '/blog' for example.

If you have time to look through Ghost's initialisation process and suggest how we can get from A to B, that would be super helpful, and very much appreciated. I know that @hswolff is very keen to get this in, so perhaps he has some more specific questions for you.

[hswolff]

I've been thinking about this problem a lot recently.

The most straightforward idea I have of solving this issue involves a fair amount of refactoring, centered around the idea that every module within Ghost must be instantiated to be used as opposed to just being required to be used. This would come in hand with a Ghost 'class' that would be instantiated for each middleware instance, and which would in turn instantiate each module that Ghost requires for a working application instance.

Would love to hear some preliminary thoughts on that architecture idea @ErisDS. How we get to there will be done slowly, migrating/refactoring one module at a time (i.e. models, config, basically every directory is what I think of as a module).

[dougwilson]

If you have time to look through Ghost's initialisation process and suggest how we can get from A to B, that would be super helpful, and very much appreciated.

Yes, I definitely need to do this.

[sebgie]

👍 to everything @hswolff said. One thing to keep in mind is that we have killed the ghost singleton just a few weeks ago and we should beware from bringing it back to life again.

[martinsookael]

Making

npm update ghost

or

npm update casper

would be much more convenient then the current updating process is:
http://support.ghost.org/how-to-upgrade/

[ErisDS]

@martinsookael It is totally possible to setup Ghost that way right now if you want to. This issue is about making Ghost work as express middleware so that you can use it as part of a larger express application.

[ErisDS]

@hswolff I know we spoke about this in IRC, and we now have #3815 merged which is kind of step 1 towards making this work.

I don't disagree with any of what you have described, in fact it sounds a lot like what is described in #2182 and it definitely needs to be done. What I'm not sure about though, is why it is a prerequisite to exposing Ghost as middleware?

Do you feel the approach in #3849 will have problems unless this is done?

[anyong]

Any updates on this in the last 8 months? Is it working? Dead?

I will be looking to set up a blog in the very near future and would like to simply integrate it in my existing site, as many have mentioned above.

Thanks!

[JoshWillik]

@anyong I'm not sure where the official support stands at this point. I lost track of how the pull request from @gleneivey was doing.

I would like to point out, however, that I already have this working at my own website with a small amount of patch-code. https://www.joshwillik.com/blog/

If you need any help getting it working in your own, feel free to contact me at the email on my Github profile and I'll do my best to help you through it.

[anyong]

@JoshWillik first of all that is some impressive response time, thank you!

Second, and perhaps more importantly, the link in your latest post here from 8 months ago where you've presumably got that patch code is broke. :(

[JoshWillik]

@anyong The way I proposed is here.
https://github.com/JoshWillik/Ghost/blob/middleware/core/middleware.js

It never got much traction, but it is a functional idea.

I suspect this code will no longer work exactly as is, but it shouldn't be
too hard to patch it up.
On 4 Jul 2015 10:32 am, "anyong" notifications@github.com wrote:

@JoshWillik https://github.com/JoshWillik first of all that is some
impressive response time, thank you!

Second, and perhaps more importantly, the link in your latest post here
from 8 months ago where you've presumably got that patch code is broke. :(

—
Reply to this email directly or view it on GitHub
#827 (comment).

[gleneivey]

My pull request is still valid. I've slowed down with how frequently I rebase onto master, so at any moment it may not be a clean merge, but so far the changes in (the relevant areas of) Ghost haven't been significant enough to turn merging into rewriting.

I've got it running in a number of places, but installed it here https://myou.pub/b less than two weeks ago. (Also, in this installation I'm using a variant of Casper with Disqus integration--vanilla based on the standard Ghost instructions.)

[ErisDS]

Integrating Ghost into an express app is already a possibility. The open PRs and this issue now reflect an 'improvement' to this, which makes it possible to use Ghost as middleware, rather than as a mounted express app.

I've really been struggling to spend time reviewing the PRs, because this issue has a major design issue and the consequences require a lot of thought - the positive impact is minimal (mount as middleware instead of app) but the negative impact is hard to judge - will this prevent us from changing other things in future?

This issue has been around for a very, very long time and it definitely would be preferable to resolve it. Here's a curve ball question:

Is making Ghost work as middleware even the right thing to do and why?

Ghost works right now as a mounted App - so you get a whole app including error handling mounted at a particular route. The main difference (I think) with this is that if Ghost doesn't match the route you get Ghost's 404 page rather than Ghost passing back to your own app?

[JoshWillik]

In my (possibly incorrect, probably biased) opinion, if you want to provide a way to include ghost as a subsection of a larger site, middleware is correct method.

Control

When you have an existing application, the top-level code must be in absolute control of the way the application functions.

The way the nodejs/express community has informally standardized the way modules fit into a larger system takes the following pattern, with very little variation.

var parentApp = require( 'express' )()
var api = require( 'api-module' )
var fileUploadModule = require( 'file-upload' )
var notFound = require( './404-handler' )

parentApp.use( '/api', api( <options> ) )
parentApp.use( '/files', fileUploadModule( <options> ) )
parentApp.use( notFound )

parentApp.listen( 8888 )

The thing to notice here is that all functionality happens at the request of the parent application. The parent dictates which route the child will use, it dictates when the application will start and stop.

As far as I can read from the documentation, the currently proposed way to mount a ghost blog is as follows.

var ghost = require( 'ghost' )
var parent = require( 'express' )()

ghost( <options> ).then( function( blog ){
  parent.use( blog )
  parent.listen(8888)
})

This method has 2 huge flaws and a big flaw.

The first is that the parent application loses all control over what happens to the request after it enters ghost's domain, because the request will never exit ghost's domain. Ghost will send out a 404, no matter what the parent wants. This is annoying for a hobby blogger, and unacceptable in more enterprise settings. The parent has lost control of the request path.

Secondly, the parent no longer decides how the application starts and stops. Ghost has hijacked the boot process and forced the rest of the application to bend to the way it does things. It is no longer a well behaved child.

Thirdly, even if we were to accept that it's ok for ghost to hijack the boot process, this will confuse beginners and tinkerers who don't fully understand promises and asynchronicity. You'll see baffled and frustrated users who don't understand why

var app = require( 'express' )()
require( 'ghost' )().then( function( blog ){
  app.use( blog.rootApp )
})

app.use( <thing1> )
app.use( <thing2> )

appends middleware in the seemingly wrong order.

Solutions

In my view, there are two ways forward.

If you want to provide the ability to mount ghost at all, it must be as middleware. This is the standard the community has made, and it should be followed unless we have good reason to ignore it (which we don't).

If you dislike the additional complexity this adds, move ghost-as-middleware into a side project.
As a proof of concept, I've published an NPM module and a repo that show this working pretty much out of the box.

I've left the name ghost-middleware open in case you want to pursue something similar.

[gleneivey]

That's a really good question. When I first set out to embed Ghost in a web app, using it as a mounted app didn't work, either, so going with a middleware interface didn't need really strong motivation.

Aside from being able to better integrate the app's over-all error handling, I think the bit thing that's lost by going with the sub-app is going back to config.js instead of a hash passed in from the parent app, and it seems like this is unrelated to middleware (could be passed in when the module's entry point is called to get ghostServer).

I much prefer config via environment variables, which I'm currently using to build a config hash with environmental.

If things stay as-is and I reach a point where the rebase to integrate my changes on top of a new Ghost release is too hard, I'll probably change over to mounting Ghost as an app and make some kind of env-to-config-file widget.

[ErisDS]

With regard to config management - there's a separate issue (#4731) which also has a couple of languishing PRs open against it. I think the config issue applies to several use-cases, so it needs solving separately.

The one thing that stands out to me most of all when looking at #4101 is that it seems to touch areas that I wouldn't expect. Maybe it'd be worth explaining that in more detail on the PR? I'm thinking perhaps resolving the config issue first (and perhaps splitting out one or two other details from the PR) might help to reduce the surface area of the changes needed to make the middleware PR work. The smaller the footprint, the easier it is to understand the consequences and merge with confidence 😉

[gleneivey]

I'm not sure which areas are the "unexpected" ones, but the things that caught my eye with a quick scan are:

• the config error checks were refactored so they could be applied to an incoming config hash, including removing the actual file read from validate().
• there were a lot of repercussions from the desire to not duplicate the path to Ghost within the broader app's URL space in the config hash, and to have it only in the .use() call that installed Ghost into the main app's middleware chain:
  • ghostConfig.url, and therefore baseUrl in most places, is not available when the app initializes, so there are a number spots where repeatedly used code has guard clauses
  • in particular, this is why the site map stuff (xmlrpc) is included in the changes
• I introduced the ENABLE_GHOST_LOGGING environment variable because my production environment isn't actually called "production". I actually think this is a broader problem: Ghost should have separate on/off switches for environment-specific behavior, or a way to configure the environment name strings or something.
• the functional tests had to accommodate URLs that refer to either "localhost" or "127.0.0.1". This is a side effect of whether or not we got through express to the the base URL. I looked into code changes that would normalize this, but they would have been more invasive. And changing config.example.js felt like cheating. Could have not run unmodified functional tests in the middleware configuration, but that felt too open-loop for the changes.
• the change in migration_spec was an unanticipated pain, but I think the comment explains it as well as can be

Other questionable stuff?

Thanks,
glen

[anyong]

Well I'm getting ready to dive in over the next couple of days as the time has come for the blog part of my project...

I tend to agree with @JoshWillik above. Everything from parameter parsing to i18n to database links to eCommerce to single page apps and more all follow a fairly standard express protocol.

If I understand correctly, the main configuration issues for ghost are the database and mail server. The database uses knex, and the authentication backend uses passport. These are both really standard tools that I think many sites have already in use, and I'm sure there are many others who are already using or would like to simply plug in their existing database and authentication mechanisms for ghost - that's certainly what my approach is.

app.use('/blog', ghostMiddleware(ghost));

seems like a perfectly reasonable way to do it, this is pretty much how knex/bookshelf work, too. We can do our ghost config outside of app.js and simply require the configured instance and pass it to a middleware function that starts the app and passes 404s to next().

Anyway, I think it would be great if this was plug-and-play like knex/bookshelf is, so I'm happy to help now that I've got some time for it... please let me know if you've got recommendations for what to do since I'm new to the library. In any case, I'll start looking around and see if anything jumps out. Thanks!

[ErisDS]

The discussion and movement on this issue has tailed off, largely because it has been very hard to get a full understanding of what we need to change and why. I believe this is also in part because the underlying goal of the issue is effectively solved, even if it's not perfect, and that there's therefore a lot less traction on this idea than there was previously.

There is definitely a great deal of refactoring work to be done inside of Ghost's boot, server & middleware code, and I have always felt that a more natural solution to the ghost-as-middleware issue would fall out of its own accord if #2182 and #4172 were fully addressed and if some of the difficulties around configuration & logging were solved properly.

I'm going to close this issue as 'done' for the time being, and focus on ensuring there is a clear idea of how the internals of Ghost might be restructured. I believe the goal of restructuring the Ghost internals in a more sensible, reusable, composable, and 'nodey' way is higher priority and more obtainable, and I think once that work is done, if the ghost-as-middleware use case doesn't improve naturally, then we will at least be able to revisit this issue from a better position.