Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Server-Side Rendering for SEO #9938

Closed
tomdale opened this issue Dec 15, 2014 · 57 comments
Closed

Implement Server-Side Rendering for SEO #9938

tomdale opened this issue Dec 15, 2014 · 57 comments

Comments

@tomdale
Copy link
Member

@tomdale tomdale commented Dec 15, 2014

Currently, Ember apps must be downloaded in their entirety, evaluated, and executed before the end user sees any UI. Typically, most Ember apps serve a static "loading" page that is replaced by the framework when the app boots up.

This can lead to degraded experiences as the user has to wait for JS and other assets to load before they get any actionable UI. The issue is exacerbated for mobile devices, which can take up to 10x longer than the desktop to boot the app, and are often connected via high-latency mobile networks. Users in the developing world may be using slow connections and older browsers and suffer a degraded experience. Fetching the initial data payload as a separate request on these high-latency networks imposes an additional penalty.

SEO is also a problem for client-side rendered apps. Despite Google recently announcing that their search crawler would evaluate JavaScript when scraping sites, the secrecy around ranking algorithms leads many organizations to distrust relying on this new system vs. the well-understood behavior of server-rendered pages.

Additionally, crawlers or other tools like curl can't fetch content from JavaScript apps (without embedding a JavaScript engine, anyway). This ability to remix and scrape content is one of the strengths of the web.

We would like to address all of these shortcomings, and we're starting with SEO.

Thanks to the generous sponsorship of Bustle, @wycats and I will be working on booting and rendering Ember apps on the server. While we've designed Ember from the beginning for server-side rendering, the initial work will largely be infrastructural, getting all of the pieces in place for Ember apps to run statelessly on the server.

This work is a prerequisite for the more ambitious goal of rendering apps on the server, then downloading JavaScript in the background, and having it take over control once loaded. (This is sometimes referred to as "rehydrating" the client, a.k.a. "FastBoot™".)

This issue tracks the infrastructure changes needed to achieve the SEO server rendering MVP.

  • Abstract browser environment necessary for executing the Ember library (#9941)
  • Abstract browser environment necessary for booting an Ember application (#9941)
  • Incorporate tests to verify node compatibility of built output
  • Support booting application in node environments
  • Split container into code registry (stateless) and session store (stateful)
  • Remove 1-to-1 application <-> session coupling
  • Render component to string instead of DOM element
  • Render a route to string
  • API for asynchronously initializing an Application (Application.boot().then(callback))
  • API for asynchronously rendering multiple sessions using single Application (app.visit('/my-route').then(callback))
  • Audit and disable assertions that are browser specific when running in server-rendering mode
  • Build demo app showing the feature working
  • Compile Ember into a directory of CommonJS modules It turned out to not be too difficult to use the same compiled output that's sent to the browser in Node. We will revisit this later.
  • Distribute CommonJS Ember build on npm
  • Implement boot lifecycle hooks (entered loading state, application state has settled, application has rendered). These will be useful for things like analytics integration as well. Still want to get to this, but we discovered it is not on the critical path.

Expect this list to expand over time as we start to dig in.

@tomdale tomdale added the Feature label Dec 15, 2014
@jayphelps

This comment has been minimized.

Copy link
Contributor

@jayphelps jayphelps commented Dec 15, 2014

@tomdale How bout first-class ember-data store.push() so the client doesn't need to fetch the models again?

@fivetanley

This comment has been minimized.

Copy link
Member

@fivetanley fivetanley commented Dec 15, 2014

I just want to note while I am very excited, please don't add posts like "👍" or "yay!" We want people to contribute constructive feedback, but we want people to participate without getting hundreds of emails that don't have content. Thanks!

@tomdale

This comment has been minimized.

Copy link
Member Author

@tomdale tomdale commented Dec 15, 2014

@jayphelps Out of scope for the current phase but definitely something we intend to incorporate once this is solid.

@tim-evans

This comment has been minimized.

Copy link
Contributor

@tim-evans tim-evans commented Dec 15, 2014

Could we have recipes for deployment? This seems like it's a tad more complicated to deploy than 'just push to S3 and invalidate cloudfront'. (This is really awesome to see, btw)

@endash

This comment has been minimized.

Copy link
Contributor

@endash endash commented Dec 15, 2014

Seems to me you'd also want a mechanism for clearly distinguishing pre- and post-rehydrated apps. a and button elements that would otherwise be tied to UX for instance would in a lot of cases need to start out as disabled or occluded by a click capturer. A lot of the visual indications could be handled via a simple class added to the root element, though.

@stefanpenner

This comment has been minimized.

Copy link
Member

@stefanpenner stefanpenner commented Dec 15, 2014

hopefully iojs will be a real thing by then, so we don't need to use a year old version of v8...

@stefanpenner

This comment has been minimized.

Copy link
Member

@stefanpenner stefanpenner commented Dec 15, 2014

a and button elements that would otherwise be tied to UX for instance would in a lot of cases need to start out as disabled or occluded by a click capturer.

ya it will be important to not by default have a broken UI, that would be a bad trademark of these apps. Hopefully some pattern/convention will emerge so satisfy this in a sensible way.

@ryanflorence

This comment has been minimized.

Copy link

@ryanflorence ryanflorence commented Dec 15, 2014

a and button elements that would otherwise be tied to UX for instance would in a lot of cases need to start out as disabled or occluded by a click capturer.

If you build your app in a certain way, namely, like a server-app, the whole thing will still work with early clicks even before the client-side app bootstraps.

I do this in react with react-router. Instead of handling actions "inline" with a controller action, you actually make a transition to a new route, usually with query params. Whether you're opening a modal, or submitting a form, use the router and query params and your non-client-side-bootstrapped app only loses out on animations, but the state is fully renderable by the server or client.

Its a mind-shift back 5 years, but it works beautifully.

@ryanflorence

This comment has been minimized.

Copy link

@ryanflorence ryanflorence commented Dec 15, 2014

How bout first-class ember-data store.push() so the client doesn't need to fetch the models again?

Again, with react-router, I dump initial data in a script tag (like window.__PRE_RENDER_DATA__), and then when the client-side app initializes it primes the data stores with the stuff in there. Now when your app asks for the data, its already got it (and for react, this causes the HTML checksum created by the client to equal the pre-render from the server, so the DOM doesn't get thrashed, only events get added)

Big prereq is for folks to get their model hooks to work isomorphically.

@tomdale

This comment has been minimized.

Copy link
Member Author

@tomdale tomdale commented Dec 15, 2014

@tim-evans Definitely. We are still exploring how exactly deployment happens. Probably we will have an application server that you can deploy built versions intended for server-side rendering; if so, we will build this workflow into Ember CLI (potentially via an addon).

@tomdale

This comment has been minimized.

Copy link
Member Author

@tomdale tomdale commented Dec 15, 2014

@endash There's a lot to think about, definitely, and a lot of edge cases. That's why we're keeping the scope constrained to just rendering and punting on rehydration until we better understand all the cases.

@caridy

This comment has been minimized.

Copy link

@caridy caridy commented Dec 15, 2014

sidenote/rant: There is a fundamental issue when rendering the app on the server side that is usually not taken into consideration when designing the libraries, or the framework, and it happens to be an essential part of the business for many complex applications, and that is the bucket infrastructure. In most cases, bucketing new features in an app is all about UI elements, but for SPAs, it transcend the boundaries of the CSS/HTML, and a different code path are needed (e.g.: module A requires module B, but sometimes, for some users in a bucket, it should import from B-bucket123 (which implements the secondary code path).

For the client side, this is easy, you rely on a build process that produce different bundles with different code paths, and you just need to worry about delivering the right bundle, and the right set of modules, and probably a custom loader that knows how to do the right wiring for those dependencies, but on the server side, things can go south really fast if the code path can't be decided per request, forcing people to fight the framework to try to implement a bucket infrastructure. This is a problem that we have with React, and the use of singletons, etc.

@ericf I don't know if you have talked to @wycats about this. If not, maybe it is a good time to do so, to make sure that whatever solution gets implemented for the server side rendering is flexible enough to introduce alternative code path per request.

@tomdale

This comment has been minimized.

Copy link
Member Author

@tomdale tomdale commented Dec 15, 2014

@caridy Yes, please have @ericf talk to @wycats. :D We would love to make sure we understand your requirements. It would be great to have Yahoo! be a 100% Ember shop. ;)

@vanthome

This comment has been minimized.

Copy link
Contributor

@vanthome vanthome commented Dec 15, 2014

Great stuff! This is the future. As developer working on a quite big EmberJS based app I'm involved with all these issues described. A main problem we found for server side page rendering is to identify the point in time, when the page has finished rendering and the DOM can be scraped. We solve this with a cooperative approach between renderer an app as described here:
https://github.com/n-fuse/the-XContentReady-Event.
Maybe this is also relevant for pre-rendering as outlines in these plans.

@tomdale

This comment has been minimized.

Copy link
Member Author

@tomdale tomdale commented Dec 15, 2014

@vanthome We are planning to have an API for detecting when the application boot has settled very soon. It is one of the first tasks we are working on as part of this effort and should have something in a few weeks.

@ryanflorence

This comment has been minimized.

Copy link

@ryanflorence ryanflorence commented Dec 15, 2014

Another constraint to worry about is singleton state.

While ember can clean itself up of all singleton state, a lot of users won't. This can cause major issues on the server since its not single-tenant like the browser.

One idea is to allow singleton state by wrapping the entire application in a function at build time and then executing per request (with optimizations proabably). This way the singleton state is closed over per request.

The conventions of ember-cli can help a lot here since it already owns the build.

@chadhietala

This comment has been minimized.

Copy link
Member

@chadhietala chadhietala commented Dec 15, 2014

Our experimentation framework has a bucketing strategy as well. I foresee us going down the road of permutation based builds were you have a completely different application instance based on the bucketing strategy and traffic allocation.

@caridy

This comment has been minimized.

Copy link

@caridy caridy commented Dec 15, 2014

@chadhietala permutation based builds only works for the client side, it is not realistic to execute all the code (for one of the permutations) per request on the server side to keep them in isolation. Some people has suggested used vm to execute in a new context, but that might not scale for big complex apps due to the perf penalty of executing all the app code all over again per request.

@donaldpipowitch

This comment has been minimized.

Copy link

@donaldpipowitch donaldpipowitch commented Dec 16, 2014

Will this work with HTMLBars?

@zeppelin

This comment has been minimized.

Copy link
Contributor

@zeppelin zeppelin commented Dec 16, 2014

@donaldpipowitch I'd say absolutely!

@wycats

This comment has been minimized.

Copy link
Member

@wycats wycats commented Dec 16, 2014

@zeppelin indeed. It's being built in terms of HTMLBars :)

@egaba

This comment has been minimized.

Copy link

@egaba egaba commented Dec 16, 2014

Big players making big plays. Keep up the great work, guys!

@ericf

This comment has been minimized.

Copy link

@ericf ericf commented Dec 17, 2014

@tomdale @wycats the checklist looks good to me!

There is one area for this phase of work that I think should be added to the checklist, and that's around short-circuiting. Since the requests to the server are stateless there could be some good opportunities to removing auto-wiring and observable bindings, because setting those up can be too expensive for the server and are not likely to be used server-side.

Related is Component lifecycle and making it possible for devs to avoid costly operations that won't be needed server-side (like React, but I'm sure Ember Components already have something like this), similar to the work of avoiding DOM operations. This, with general short-circuiting might allow for the server to avoid executing entire code paths that aren't required in handling the current request.

What's the plan for the rendering out the <html>, <head>, and <body> elements? I'm assuming that the Ember app will deal with everything inside some container <div> in the <body>

Is this within scope here, or would it be up to the dev to provide the wrapping HTML around the app's visible content?


How bout first-class ember-data store.push() so the client doesn't need to fetch the models again?

Again, with react-router, I dump initial data in a script tag (like window.PRE_RENDER_DATA), and then when the client-side app initializes it primes the data stores with the stuff in there.

@jayphelps @rpflorence we do this using express-state which is built on serialize-javascript. And a team at Yahoo has wrapped-up this idea in a tidy way for their Flux implementation, [enabling stores to dehydrate/rehydrate](https://github.com/yahoo/fluxible-app#dehydrationrehydration automatically).

@stefanpenner

This comment has been minimized.

Copy link
Member

@stefanpenner stefanpenner commented Dec 17, 2014

Since the requests to the server are stateless there could be some good opportunities to removing auto-wiring and observable bindings, because setting those up can be too expensive for the server and are not likely to be used server-side.

Potentially, although some users unfortunately do rely on this even for basic rendering. Maybe some good patterns exist to help users prevent this.

@JamesMGreene

This comment has been minimized.

Copy link

@JamesMGreene JamesMGreene commented Dec 20, 2014

\o/

@duizendnegen

This comment has been minimized.

Copy link

@duizendnegen duizendnegen commented Dec 22, 2014

Super awesome. I'd say this makes a logical coupling to lazy-loading Ember apps as well, or is that just wishful thinking?

@duizendnegen

This comment has been minimized.

Copy link

@duizendnegen duizendnegen commented Feb 14, 2015

Prerender.io worked their magic for me, also available as an open source package

On Sat, Feb 14, 2015 at 12:46 AM, Nathan Palmer notifications@github.com
wrote:

@Fed03 @taras in practice I've had little success with google crawler on an ember app. Seems like it may only work on simpler ones. I admit I haven't spent much time digging into it.

Reply to this email directly or view it on GitHub:
#9938 (comment)

@Yankovsky

This comment has been minimized.

Copy link

@Yankovsky Yankovsky commented Mar 28, 2015

@tomdale Server side rendering is a great feature, it is a game changer! Can you estimate when it will be ready?

@mko-io

This comment has been minimized.

Copy link

@mko-io mko-io commented Mar 28, 2015

@Yankovsky as Tomhuda said in emberconf all thing will be released in June this year keep tuned.

@tsteuwer

This comment has been minimized.

Copy link

@tsteuwer tsteuwer commented May 3, 2015

My only concern here is that you're only building this for node. Not everyone uses node with ember. There's ruby, php, and asp, to name a few.

@dschmidt

This comment has been minimized.

Copy link
Contributor

@dschmidt dschmidt commented May 3, 2015

Just because you build your REST API with PHP, ASP, Ruby or what not, it does not mean you can't serve Ember with node.js

@tsteuwer

This comment has been minimized.

Copy link

@tsteuwer tsteuwer commented May 3, 2015

Yes, but making it a requirement is my concern.

@aprescott

This comment has been minimized.

Copy link

@aprescott aprescott commented May 3, 2015

What's the root concern you see with having a node requirement for this feature? What do you see as the actual problem?

@topaxi

This comment has been minimized.

Copy link
Contributor

@topaxi topaxi commented May 3, 2015

@tsteuwer how would you run javascript code on ruby, php and asp? There is no way, emberjs could be run on those runtimes (without the obvious enormous work of a maybe possible transpilation)...

@tsteuwer

This comment has been minimized.

Copy link

@tsteuwer tsteuwer commented May 3, 2015

I work for a large company and I'm in the prototyping phase of using ember. We won't be converting over our consumer facing portion for about 6 months if we find that ember will work well with our site. Currently we have a large framework which builds out the HTML payload the user will see on load using php, then we have objects which handle the interaction and moving forward in js. I don't see the Arch's agreeing that we must install node and use it to render our HTML so that we can get seo when we have all the tools currently in php. Now, if it were a php plugin as well, I can see them understanding and allowing its implementation. However, having to install another runtime on top of what we already have and use may be an entirely different story.

@tsteuwer

This comment has been minimized.

Copy link

@tsteuwer tsteuwer commented May 3, 2015

@topaxi, thats exactly it. So ember users will only get the benefits of seo only if they use node? That's basically the root of the issue.

@NuckChorris

This comment has been minimized.

Copy link

@NuckChorris NuckChorris commented May 3, 2015

@tsteuwer With ember-cli we should all already be using a static index.html (generated by node, it's already in your stack!) which is served by your web server, not by PHP.

The addition of Node to prerender the ember app makes sense and is an easy integration. What you're asking for (prerendering in PHP) is effectively impossible and honestly makes no sense.

Your server side code has one purpose in Ember: providing a REST API.

@tsteuwer

This comment has been minimized.

Copy link

@tsteuwer tsteuwer commented May 3, 2015

@NuckChorris I completely agree. However, lots of us don't get to choose the stack we get to work in. I'm all for node, but u fortunately we've had a lamp stack for the past 15 years and wouldn't be able to get that switched over. I may just have to find aa workaround. Is it possible to point controllers at specific ids or tags and let ember initiate on doc.ready à la angular?

@mixonic

This comment has been minimized.

Copy link
Member

@mixonic mixonic commented May 3, 2015

@tsteuwer I'm willing to bet your company also uses memcached, redis, or nginx and thus C as part of their stack. Despite this you don't think of yourself as running C, you think of these servers as utilities you put in place and forget about.

The goal for FastBoot is largely the same. You need to write Ember code, but the intent is that you do not need to write Node.js as such. It just happens to be that your Ember code works on the server as a utility you can point nginx at.

You will need to run that utility as part of your stack though, and if it is rejected (just like they might reject MongoDB or any other server) then you will need to find an alternative solution.

I don't see the Arch's agreeing that we must install node and use it to render our HTML so that we can get seo when we have all the tools currently in php.

And I don't suggest that you make that the pitch. FastBoot allows you to write your application once and run it for offline clients, mobile clients, and web apps as we think of them today. It encourages your organization to adopt a service-oriented architecture with a single API used by all clients. This pitch takes you into a discussion about the benefits of a JavaScript application and API architecture, and I hope that the benefits of those strategies outweighs installing a new daemon and language.

This might not be a trivial discussion, and might represent a change in direction for your organization. I don't suggest it is a small thing.

Lastly, if your business does not require an SEO strategy then FastBoot is not needed. The current deployment solutions will suit you just fine. In fact even if you do need SEO, there are a number of other strategies used by apps today (noscript, prerendering, dynamic server-side HTML) that will still be viable.

@NuckChorris

This comment has been minimized.

Copy link

@NuckChorris NuckChorris commented May 3, 2015

Is it possible to point controllers at specific ids or tags and let ember initiate on doc.ready à la angular?

I'm no expert but I'm pretty sure Ember isn't built to run part of an app. it's huge and all the concepts inside it (like the Router) don't really make sense if it's just managing a chunk of DOM. It's a big opinionated framework for your entire application.

@tsteuwer

This comment has been minimized.

Copy link

@tsteuwer tsteuwer commented May 3, 2015

@mixonic, that's a very valid point. Thanks all for your help

@tomdale

This comment has been minimized.

Copy link
Member Author

@tomdale tomdale commented May 20, 2015

@tsteuwer We want to support as many platforms as possible. However, FastBoot requires running your Ember.js application on the server, which is, of course, written in JavaScript. That means you must have a JavaScript runtime.

We hope that the community will build many packages to make integrating into existing stacks very easy. Ultimately, though, there is a hard requirement for a JavaScript runtime. You just can't evaluate JavaScript without one.

@rwjblue

This comment has been minimized.

Copy link
Member

@rwjblue rwjblue commented Aug 23, 2015

@tomdale - I believe that this issue is completed (all the checklist items seem to be done), and we are just waiting to "go" the feature (hopefully for 2.2). Can you confirm and close?

@Master244

This comment has been minimized.

Copy link

@Master244 Master244 commented Sep 17, 2015

I guess a demo app is pretty important in this case. I'm stuck on the problem getting Facebook to see my meta tags in my head that change dynamically when I go to a new news item. I got this al working but then I noticed idd FB does not run any Javascript and just defaults to my standard og:description etc so i'm kinda stuck. And should probably be looking for something like this. Where a working demo would come in very handy :)

@duizendnegen

This comment has been minimized.

Copy link

@duizendnegen duizendnegen commented Sep 17, 2015

@Master244 https://github.com/zipfworks/ember-prerender / www.prerender.io combined with https://www.npmjs.com/package/ember-cli-meta-tags can help you out with that, while we're waiting for FastBoot to land
The packages feature some demos themselves, should be enough to help you on your way.

@wcurtis

This comment has been minimized.

Copy link

@wcurtis wcurtis commented Sep 17, 2015

To add to @duizendnegen's suggestion, the ember-cli heroku buildpack supports prerender.io config https://github.com/tonycoco/heroku-buildpack-ember-cli#prerenderio. That's my current setup and works fine (still can't wait for fastboot though!).

@Master244

This comment has been minimized.

Copy link

@Master244 Master244 commented Sep 21, 2015

@duizendnegen dankjewel :), that will defiantly help me on my way. I did find prerender.io and http://www.emberjsseo.com/ . I was already setting this up to test it out with Docker ill post my findings here if someone wants to use my Docker setup and flow.

@Fed03

This comment has been minimized.

Copy link

@Fed03 Fed03 commented Sep 21, 2015

@Master244 let us know!

@JamesMGreene

This comment has been minimized.

Copy link

@JamesMGreene JamesMGreene commented Oct 27, 2015

Yes, please! 👍

@mariendries

This comment has been minimized.

Copy link

@mariendries mariendries commented Nov 16, 2015

Is there an estimated landing time for ember Fastboot?

@rwjblue

This comment has been minimized.

Copy link
Member

@rwjblue rwjblue commented Nov 16, 2015

The Ember internal feature that fastboot takes advantage of (the new visit API) is enabled by default on canary builds and is planned to be in Ember 2.3.0.

This will allow ember-cli-fastboot to continue to improve stability and eventually graduate to a 1.0.0 version itself.

@rwjblue

This comment has been minimized.

Copy link
Member

@rwjblue rwjblue commented Nov 16, 2015

@tomdale - Closing this since ember-application-visit is enabled now. Please reopen (and update checklist above) if there are additional items you would like to track in this issue.

@rwjblue rwjblue closed this Nov 16, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.