Support the include directive on GitHub

[mojavelinux]

We'd like to find a way to support includes on GitHub in a way that works within the boundaries of the security requirements. This issue is intended to plan for and track the progress on this upstream request.

[ciarand]

Is there any way to preprocess include directives?

i.e. a project with these files:

// foo.adoc
foo

// bar.adoc
bar

// README.pre.adoc
include::foo.adoc
include::bar.adoc

And running something like this:

$ asciidoctor <some magic flag> README.pre.adoc -o README.adoc

Will generate a file named README.adoc that looks like this:

foo
bar

Is this a thing that exists?

[mojavelinux]

Indeed. I wrote a coalescer script and put it in the extensions-lab repository.

See https://github.com/asciidoctor/asciidoctor-extensions-lab/blob/master/scripts/asciidoc-coalescer.rb

Does that suit your needs? This might be nice to have either as an cli in Asciidoctor or as a separate project.

[ciarand]

This is fantastic, thanks @mojavelinux.

[MarkCallow]

What are the issues with the way the include directive currently works and GitHub's security requirements? It is hard to make any useful suggestions in the absence of that background.

My use is for including fragments of some of my source files (.c, .cpp, .frag, .vert, etc.) in docs such as tutorials that are describing the code. I would really like to not have to manually regenerate documents anytime the code changes.

Could the coalescer script be run in a push hook on GitHub or would that violate their security requirements too?

[miltador]

Any updates on this?
Lacking support of includes on GitHub limits asciidoctor usage and maintaining a lot of documentation is harder than it could be 😫 . Wonder if I can help with this.

[mojavelinux]

While I absolutely agree that not having the include directive on GitHub is a huge pain, there is at least one approach that makes it somewhat workable.

What you do is have a CI job come through an run a script that expands include directives so you get something like this in the file afterwards:

ifndef::env-github[]
include::file-to-include.ext[]
endif::[]
ifdef::env-github[]
contents of include file get added here
endif::[]

The part inside of ifdef::env-github[] would be auto-generated and not something the author would ever touch.

(Once the 1.5.6 upgrade happens, you can reduce the ifndef clause to a single line).

Again, this is far from ideal, but at least it would allow you to leverage the include directive and still have it display properly on GitHub. But you would need to set up an automated task that expands the include files in place.

[mojavelinux]

Could the coalescer script be run in a push hook on GitHub or would that violate their security requirements too?

That's definitely allowed and exactly what I had in mind.

[miltador]

What if to implement a limited include::s mode which will allow to only include other asciidoc files with aggressive bounds to the current working directory so user cannot do infinite ../.. or / to other directories? And maybe some other rules, so it can be considered to be turned on in safe mode?

I know that including any file can be a security problem, but including other asciidoc files that are rendered the same way should not be a problem. Github is rendering any single asciidoc file anyway, why not to render asciidoc inside asciidoc?🤔

[mojavelinux]

The custom include processor enables us to do that exactly. The problem we're currently dealing with is that GitHub has not yet shown a willingness to discuss this. I am ready for that conversation any day.

[datainvestor]

Any change this might be implemented in near future?

[mojavelinux]

This is not something we have any control over. It's up to GitHub to enable it. We've given them the code. Now it's on them to enable it. See github/markup#1095