Devise way to transclude markdown content from other repos

[crandmck]

Figure out how to deal with content that we currently "transclude" (dynamically include) from other GitHub repos. Typically, these are example repo README files. We use a special Confluence macro to "single source" this content; we can do something similar with a Jekyll plugin, but there are wrinkles.

This is part of Proposal to make LoopBack documentation open source.

Related notes:

• http://jekyllrb.com/docs/plugins/
• https://www.sitepoint.com/jekyll-plugins-github/
• https://help.github.com/articles/using-jekyll-as-a-static-site-generator-with-github-pages/
• https://github.com/crandmck/crandmck.github.io for POC of remote markdown plugin.

[crandmck]

Unfortunately, GitHub Pages only supports a few Jekyll plug-ins. For security reasons, you can't add other plug-ins such as the remote markdown plugin.

And we don't want to have to use Jekyll as a static site generator and check in the _site directory.

The alternative is to have a script/program to get the READMEs via the GitHub API, then check in those markdown files. I've got a POC at https://github.com/crandmck/get-readmes. To make this work, I still need to add the front-matter; but the basic technique works. See http://expressjs.com/en/resources/middleware.html, which uses this technique.

[bajtos]

Unfortunately, GitHub Pages only supports a few Jekyll plug-ins. For security reasons, you can't add other plug-ins such as the remote markdown plugin.

@crandmck could you please reach out to support@github.com and ask them to support our use case? It won't solve our problem right now, but it may make our life easier in the future.

The alternative is to have a script/program to get the READMEs via the GitHub API, then check in those markdown files.

It makes me wonder if we can automate this somehow. For example, list all projects that we are sourcing README files from in this module's dependencies/devDependencies. Then use CIS infrastructure to trigger a CI build whenever a dependency changed.

On an unrelated note: now that we are keeping all documentation in markdown and git, perhaps it's time to start moving bits of documentation back to individual projects? For example, the documentation for loopback-sdk-angular-cli can stay in loopback-sdk-angular-cli's README.md. That way our packages will have useful documentation on npmjs.org too.

[crandmck]

Thanks @bajtos I actually have a solution for this, though it's not quite as convenient as using the "remote markdown" plugin. I'm currently using a similar method in expressjs.com to pull in READMEs of the middleware modules in the expressjs org. I just got it working last night and populated https://github.com/strongloop/loopback.io/tree/gh-pages/_includes/readmes, and I'm starting to fold these into the docs, but not yet done.

Here are the basic instructions: https://github.com/strongloop/loopback.io#getting-external-readmes.

I'm using https://github.com/strongloop/get-readmes to periodically pull down all the READMEs we need into the loopback.io repo and then include them into the docs. The main downside (vs. plug-n) is bit more up-front work, and having to periodically run the script to update the README file, but I think it should work. I'm about 2/3 through implementing it for this repo. I'll update here when done.

It makes me wonder if we can automate this somehow. For example, list all projects that we are sourcing README files from in this module's dependencies/devDependencies. Then use CIS infrastructure to trigger a CI build whenever a dependency changed.

That would be ideal, and if we can figure it out, great! In the meantime, simply running the script periodically should work OK, at least as an initial solution. Who would be the best point of contact for me to discuss this with?

On an unrelated note: now that we are keeping all documentation in markdown and git, perhaps it's time to start moving bits of documentation back to individual projects? For example, the documentation for loopback-sdk-angular-cli can stay in loopback-sdk-angular-cli's README.md. That way our packages will have useful documentation on npmjs.org too.

Yes, I think that makes sense in the long-rung. Right now, I've got way too many other priorities just to launch the new doc site, and I don't that is a critical gating factor. But--assuming we solve the general problem above satisfactorily--we should look into this post-launch.

[crandmck]

Update: I create a new "readme" template specifically for transcluding README content (example repos are biggest example currently, but once we get this ironed out, we can use it for any relevant repo).

Here's an example: http://loopback.io/doc/en/lb2/Tutorial-access-control.html
Source: https://github.com/strongloop/loopback.io/blob/gh-pages/pages/en/lb2/Tutorial-access-control.md
Copied README: https://github.com/strongloop/loopback.io/blob/gh-pages/_includes/readmes/loopback-example-access-control.md

NOTE: There are currently some issues with how Jekyll renders the markdown because the Kramdown parser is more finicky than GitHub's. The biggest "gotcha" is that in headings, there has to be a space between the ##, ### (or whatever) and the heading text. So, we're going to have to revise most of the READMEs accordingly :-(

[bajtos]

It makes me wonder if we can automate this somehow. For example, list all projects that we are sourcing README files from in this module's dependencies/devDependencies. Then use CIS infrastructure to trigger a CI build whenever a dependency changed.

That would be ideal, and if we can figure it out, great! In the meantime, simply running the script periodically should work OK, at least as an initial solution. Who would be the best point of contact for me to discuss this with?

@rmg, but he is quite busy right now working on important CI improvements, at least that's what I've been told.

We would like to get @gunjpan more involved with all things CI-related, this may be a good starter for him.

[bajtos]

NOTE: There are currently some issues with how Jekyll renders the markdown because the Kramdown parser is more finicky than GitHub's. The biggest "gotcha" is that in headings, there has to be a space between the ##, ### (or whatever) and the heading text. So, we're going to have to revise most of the READMEs accordingly :-(

@strongloop/loopback-dev Please keep this in mind when writing markdown files and reviewing pull requests, to save us pain later in the future.

@crandmck what happens with files missing the space? Will Kramdown print any warning/error? I am wondering how much effort will be required to write a simple linter/checker tool that can be run as part of CI to reject pull requests containing changes with incorrect markdown formatting.

[crandmck]

what happens with files missing the space? Will Kramdown print any warning/error? I am wondering how much effort will be required to write a simple linter/checker tool that can be run as part of CI to reject pull requests containing changes with incorrect markdown formatting.

What I've seen is it just rendering the #Heading line as text, rather than formatting with <h1>... or whatever. I didn't see any error messages, but it's possible there might be some in debug mode or something, or it could just silently fail because it doesn't even recognize that it's intended to be md formatting. It's probably worth looking into, but I intend anyway to go through all the REAMEs initially to fix these kinds of problems.

[superkhau]

Please keep this in mind when writing markdown files and reviewing pull requests, to save us pain later in the future.

I don't agree with this, we should just figure out how to integrate a linter as you suggested to check for us (instead of relying on our memories).

[crandmck]

Using a linter for the READMEs is probably a good idea.

According to the CommonMark spec*, you MUST have a space (or newline) between the # characters and the text in a heading. So a bunch of our READMEs are malformed. Apparently npm is using this for their parser, so pages that don't follow it won't layout properly on npmjs.com. That doesn't apply for example repos, but of course in general it's important.

* The closest thing there is to a "real" markdown spec.

[superkhau]

I think CommonMark is a fine standard to follow, let's get on it. ;)

[bajtos]

I don't agree with this, we should just figure out how to integrate a linter as you suggested to check for us (instead of relying on our memories).

Let's do that!

remark-lint (package, docs) looks like a great tool for this purpose.

@superkhau what's the plan?

[superkhau]

@bajtos I think we should spike first: #49. The outcome of that spike should be a list of concrete tasks. The only thing left is we assign the task to someone, but I'll leave the prioritizing on your plate. ;) I'll just put it on TOB for now, not sure if it's major. Discuss more during +1 tmr.

[crandmck]

The basic work in this repo is done. We got the example READMEs, but still need to do #53 for the connectors.