Persist to GitHub

[tomcrane]

precondition - user has a GitHub acct

• Open the manifest editor in a browser - either your own hosted version of it, or our hosted version.
• Pick the "log in with GitHub" option
• In the manifest editor UI, pick repo from list or create a new repo. This repo must already have a branch configured for gh-pages; the manifest editor reads/writes in that. If you create the repo from the editor, it will configure it for gh-pages automatically.
• Now you are in a persistence environment where when you create or update manifests, they get saved straight to GitHub, and are immediately hosted with HTTPS and CORS for you. You can see them, see the structure, share them, load them into a viewer...

Editor uses GitHub API with your creds (it does NOT persist these anywhere!) to list repos, create repos, browse directories, get content, push commits etc. Is that possible? No local Git, obviously; a "save" is a commit and a push.

Does https://github.com/octokit/request.js allow for that?

This means it's fine for people to use our hosted manifest editor. We don't incur any costs or bandwidth for user-generated content, people can get results immediately, see them in viewers. They have hosted IIIF straight off the bat.

There are all sorts of value-adds, IIIF Codepen equivalents, Netlify wrapper sites etc that could be done on top of this. But just JSON, just manifests in gh-pages, is a killer app in its own right, let alone being the basis for all sorts of more interactive/feature rich stuff later.

Immediate use case - IIIF teaching, where just seeing the hosted JSON is enough.
Immediate use case - anyone can build and host a manifest.

Note - this is not about image (or other media) content. Although you could save static images, AV files etc too.
The simple case is where the Image APIs are somewhere else, you're picking existing ones.

The manifest editor might ALSO have a "save to DLCS" feature, or a Forager browser feature for content resources, but these are, crucially, independent of the persistence of Manifests. You might be using Image API endpoints that you set up in the previous day's lesson, or third party image resources, or in fact anything.

[tomcrane]

Further comment

This is not for developers (although developers might still love it too).
Anyone can create a GitHub acct.
Once you have that, from inside the editor it just feels like browsing and saving to a file system.
You just get an HTTPS address for each "file" as a bonus.

Further thought on the gh-pages...
While that's better left on by default, some use cases might not want the manifests made public, so it doesn't have to be on; they just have to find some other means of serving the manifests. Which would be up to them.

[tomcrane]

(via Glen Robson)

http://audiannotate.brumfieldlabs.com/ works a bit like this. This integrates with GitHub for storage, but it is not an entirely client-side app. https://github.com/hipstas/AudiAnnotate

[tomcrane]

Further thoughts on this.

Use case: institution has different users who are allowed to work on different manifests in different places.
The initial scenario description above assumes that the user logs in with GitHub creds and is then working in their own account's repo(s). But there's no reason why this has to be the case!

If University of X wants to let user A work on some things, and user B work on other things, they can manage that by managing GitHub permissions on different repos in their own org on GitHub.

University of X's hosted manifest editor is configured to support GitHub login, and be associated with one or more orgs, and to list certain repos within those orgs (to avoid giving the user a choice of any repo they have access to as a working location).

Some users have write access to github.com/univ-of-x/project-a and others have write access to github.com/univ-of-x/project-b. Both of these are configured as offered working locations to users, but the manifest editor only makes available working locations that the user has GitHub permissions for.

This offloads both storage and user management to GitHub.
And also offers hosting via gh-pages, if that is also configured.

[tomcrane]

Another cool example of GH persistence:
https://twitter.com/niqui_o/status/1357462657897439233

[tomcrane]

See spike (investigation) work required here:
#3

Github persistence on its own can be frustrating as gh-pages caching prevents a rapid save-preview-modify cycle.
But serving the work in progress from the local filesystem for preview, and committing to Github less frequently, could get round this... if it works.

[sammeltassen]

This also relates to the manifest & canvas & annotation ids generated by the editor. Is it possible to get the url base for the deployed json through the GitHub api, and then use this for the id?

Question: why is gh-pages essential for publishing the jsons, as it's also possible to work with the raw json source from GitHub with HTTPS and CORS? Main benefit of gh-pages would be custom urls, no?

[stephenwf]

Yes this factors into how we store and mutate the IIIF. We should be able to take a manifest and be able to relocate its identifiers and have levels of customisation for naming conventions. This is something that should be baked into the tools, and not an afterthought.

[tomcrane]

Main benefit of gh-pages would be custom urls, no?

Does using raw.githubusercontent.com get round the caching problems of github pages?

https://raw.githubusercontent.com/tomcrane/iiif-collector/master/objects/short-article-annos.json

I notice that this doesn't include a content-type in the response, too.

We still have the problem of id needing to match the served URL.
And it would be good to have the possibility of preview without commit+push.

[tomcrane]

A completely different approach to this is to provide, as a service, an application that:

• Bridges your GitHub operations to act as a proxy to the GitHub API, bypassing some browser security issues
• Accepts a POST of your IIIF manifest or collection and promises to host it for a brief amount of time (a few minutes - enough to preview it)

This service could mutate the id to match the location it is serving from.

That POST mechanism is present in the current manifest editor and used for CMS integration.
We're trying to get away from that and do everything via GitHub so this might be a step in the wrong direction.

[jpadfield]

I think GH persistence is great - but it would be good to ensure the system could also just work with a local file structure for implementations within institution. But for an open service the solution of save to GH works very well.

Might it also be interesting to look at the use of PIDs such as https://w3id.org/ - which could be set up to allow redirects to GH resources but then provide non GH URLs ....

[sammeltassen]

Maybe an elephant in the room, but: isn't the local file structure more important than GitHub persistence? Because GitHub offers great desktop tools already to work with local repos. And this is how I would prefer to work: local clone of repo, opening a file in the manifest editor, previewing in another tab (using the same file api–and this might be the tricky part), pushing the commits when happy.

[tomcrane]

This is a key question. Is it actually possible though?

Here's the manifest editor side of the problem:

https://googlechromelabs.github.io/text-editor/
(explained at https://web.dev/file-system-access/)

So that side is clearly possible - the user can open local files and save them back.

If I edit a JSON file my-manifest.json in there, how do I make it available on http://localhost:xxxx/.../my-manifest.json for previewing anywhere I like? (Actually we'd want it on https...)

If it's in a github repo the user can check it in and preview it via gh-pages or GitHub raw as above.
But we would like to be able to preview without git push, or not require Git(Hub) at all.

We could side-step this problem completely by telling the user that they have to run their own web server for the local file system. Just provide one-liners - "do whichever of these you prefer: https://medium.com/sweetmeat/simple-one-line-local-http-servers-8adb57d93ec3"

But... that's a big jump for casual users.

Is this a mode?

• I'm running a local web server, the root is [...]
• I'll be pushing to GitHub for save and preview (and the app can do this itself, on your behalf, ideally)
• ???

Is the third magic option possible? I'm not running a web server, I'm not previewing from GH, but somehow the manifest editor app can conjure up a resolvable URI from reading the file system, that projectmirador.org could load?

[jpadfield]

Would you "need" a local web-server - could it not be possible to output simple html pages (that call external viewer JS files) and directly includes created manifest details?

My quick test based on Simple Site code throws up CORs errors, which might kill the idea, but the viewer loaded.

[sammeltassen]

Well... you could think about a service that makes the local file available online while working in the editor. But that runs against the static workflow.

But I don't see why viewers couldn't load local files as well, like the editor? Or accept data urls like Universal Viewer already does, see:

• Make use of local file system #3 (reply in thread)

It should be feasible to implement something similar for Mirador as well and then this already solves the preview issue for the two main viewers.

(With data urls the entire json is sent to the viewer as part of the url, so no need to put the file anywhere.)

[jptmoore]

Hi, I signed up as a tester so catching up with these discussions. In terms of GH persistence I was thinking of implementing a manifest server but decided to first do an annotation server. Not sure if this type of approach is interesting to you but the server writes native git locally and users then submit PRs to share back. The web interface would ideally hide the git interaction from the user. The server is here: https://github.com/nationalarchives/miiify

[stephenwf]

Ah interesting, so the git repository could live on the server itself, with the interactions using git (opposed to githubs APIs)

[jptmoore]

Yes, so each app is bundled with frontend interface and the backend which is just a server that talks your own API but read/writes native git. So the app can be used in a distributed offline way and can be synced back through git in the same way traditional code projects work. The technology behind it is https://github.com/mirage/irmin

[stephenwf]

I've done some further exploring of this. Although having a transparent GitHub integration would be nice, I think we might run into problems with existing repositories of IIIF and repositories that mix with other content (such as static sites). I think we need to make it clear what we are doing with the GitHub repository and have the user confirm those actions.

So having the user see the repository, the branch and choose a commit message:

And then finally, a way for them (at least initially) to confirm this action through a modal:

This will be a bit of a confidence boost in the tool initially, with the second modal being optional after seeing it at least once.

The integration itself may also be tricky. Users can choose to allow "Refresh" tokens or not. That may mean that an access token expires while using the tool with no way to refresh. (requiring redirect to get a new token) Some technical challenges to overcome for sure.