Include file from disk or git URL

[racke]

This would be incredible useful for technical documentation instead of copy&paste the file everytime it changes. Examples would be programming code, configuration files etc.

[melmothx]

"Stefan Hornburg (Racke)" notifications@github.com writes:

This would be incredible useful for technical documentation instead of
copy&paste the file everytime it changes. Examples would be
programming code, configuration files etc.

Here we have to do a distinction between the main display in the
application and the static files which are built at publishing time
(PDF, EPUB, etc.) For these files, assuming that the links are moving
targets, I'm not optimist. Because you could put something in place,
publish and forget about that. If the remote file changes, the
application doesn't know about that and they would be obsoleted. Unless
the thing keeps track of them, check every day or so. Plus it breaks the
assumption that the archive is self-contained and the build is
reproducibile from any location.

However, we could have an HTML display only option, which check for
links in the body, fetch the remote, insert the content. The PDF will
have, for example, would display just file:///etc/passwd (maybe
clickable), while the site could very well check if it should display
that, fetch the content, replace it. The thing could be done via JS for
the HTML manipulation and ajax (for the actual fetching). This could be
doable and it would not be intrusive.

Would this sound acceptable?

Marco

[melmothx]

I think @perlpunk was asking something like this with regard to external images. It's basically a similar problem which could be solved in a unified way.

[perlpunk]

@melmothx right. in my use case the wiki is not seen as completely self-contained; it's a work-in-progress thing for sharing information, and outdated external resources are acceptable.

[melmothx]

@racke @perlpunk

Is a syntax like this [[file:///etc/passwd][EXT]] and [[http://remote.test/file.png][EXT]] and [[http://remote.tld/file.conf][EXT]] acceptable? The PDF and the static files would display simply an "EXT" link which is hopefully working. Then the site logic would pull the content, validate, etc.

[melmothx]

More over these lines:

• Static files, built at publish time, will have the [[link][EXT]] stuff, displayed as EXT with hyperlink
• HTML will have the ajax to pull them.
• Bookbuilder: interpolate at https://github.com/melmothx/amusewiki/blob/master/lib/AmuseWikiFarm/Archive/BookBuilder.pm#L1193 with the same logic of the ajax backend.

So you can have both the HTML and the formats with fresh content.

[melmothx]

Ok, I think the approach to the problem here is wrong.

For a rationale of amusewiki, please see https://amusewiki.org/library/rationale

This idea of external resources embedded at markup level is bad, because it just goes against the whole architecture.

However, the problem is fixable in another way, which looks saner and more secure. Because of course, here this thing would open up a whole set of vulnerabilities (think about an http://localhost/path/which/is/supposed/to/be/served/only/to/localhost and here we go, it goes embedded in a public wiki).

So, basically, you could have a script which does the following:

1. Read a user provided file with an identifier and an URL. Which at this point could be very well file:///etc/passwd as it's under the user control and out of the app.
2. In the muse file, leave some comments, like ; START INCLUDE and ; STOP INCLUDE (which are invisible in the generated formats)
3. Fetch the application git and do forced update to keep it in sync using the freshest version.
4. Scan the archive and replace with the content between the comments with the LWP->decoded_content stuff the script pulls from the identifiers, and commit the changes, if any. Bonus point: you actually keep track of the changes not just display the content at some random time. I see this as feature, of course.
5. Add a cronjob to the application which pulls this cloned git every hour or so. Example: https://github.com/melmothx/amusewiki/blob/master/script/sync-remote-archive.pl#L14 (the applications will refuse to pull non-fast-forward anyway, so it's safe).

Such script could be very well provided as example and made it generic enough for other people to use, it would require a predictable amount of time to write and test, doesn't touch the application and the underlying modules, and just feels more in line with the way amusewiki works.

Regarding the external images: either they are fetched and included in the archive at import time, and embedded into the formats, or they stay as link, as they are. Then the javascript approach could turn links to images into actual img tags to be displayed. I believe it's a matter of a dozen or two of javascript lines. This is done per site, and the javascript can be provided by the local.js file https://github.com/melmothx/amusewiki/blob/master/README#L111

[melmothx]

Is the explanation (and solution) above enough and can I close this issue?

[melmothx]

Doc added: https://amusewiki.org/library/include-remote-files

[melmothx]

Parser and compile code looks ready.

melmothx/text-amuse-compile#29
melmothx/text-amuse#63