Proposal to change artwork format and extension architecture

[jvolker]

During the work on Processing extension in the past weeks I've tested, used and extended the Openframe platform a bit. The architecture is thought-through, comprehensive and has foundations for more great features that are not fully implemented yet. That's great! On the other hand, some small parts prevent extensibility and lead to confusion in some cases. I want to show what they are and propose some changes.

Decouple artwork formats from extension

Currently, artwork formats are internally referenced to by their extensions NPM package names:

Image (jpeg, png) = openframe-image
Website = openframe-website
Video (mp4) = openframe-video
Shader = openframe-glslviewer

I think this causes two main problems.

1. Image and video formats, for example, are standardised. There is a multitude of software that can handle them. Currently, the end-user can't choose which extensions to use to display the artwork. The owner of the artwork decides on that without any knowledge of the client's setup or intentions.
2. Extensions block the namespace of an artwork format even though they only support a subset of common artwork types.

Let's take the image extension as an example. Even though it was built with great intentions (high performance on slow device) in mind, it has its limitations. I can only deal with JPG and PNG image types and low resolutions. It would be nice to have an alternative extension that can handle more image types (like GIF) and open images of high resolutions too. A VLC or Chromium based extension could do that and would be easy to implement.

But currently, there is no way to tell the Openframe client to open the artwork with a different extension. The owner/creator of the artwork decides what extension is being used to display the artwork. I also found some cases, where the artwork uploaders didn't know about the limitations by their selected extension, leaving some artworks in the public stream broken. For example, there are a few artworks that have Youtube URLs but are associated with the video extension.

Furthermore, what should an alternative image extension be named? Openframe-image would already be taken up using a name that sounds like a very general extension but is, in my opinion, rather specialised and somehow limited. Of course, the extension as it exists is important and super useful. But I think the default image extension of an artwork display would ideally support a broader range of image types. But that's a different discussion actually.

The solution: a user preferences file on the client

I propose to introduce a user preferences file on the client that allows deciding what extension to use to display artwork. It's basically what most operating systems do as well. The user chooses what application is used to open a particular file extension — usually done through right-clicking on a file in the file browser.

The file could be structured like this, for example:

Artwork format = [extension1, extension2, …]

Video (.mp4) = [omxplayer, VLC, website]
Video (.mov) = [VLC]
Video (Youtube) = [VLC, website]

Image (.jpg/.jpeg) = [glslViewer, VLC, website]
Image (.gif) = [VLC, website]
Image (Instagram) = [Instagram]
Image (Twitter) = [Twitter]

Text (String) = [Text]
Text (.txt) = [Text]
Text (Twitter) = [Twitter]

Custom (Instagram-Account) = [Instagram]
Custom (Twitter-Account) = [Twitter]

When installing an extension, the extension would register itself in user prefs file with the artwork formats it supports. It would always put itself as first of the lists having highest priority. If the user, later on, decides to change back to a different extension, they could manually edit this file at any time, changing the order of the list or deleting an extension entirely.

If artworks were then pushed to the client, the client would check in the user preferences file which extension to use to display the artwork. For example in this case Image (.jpg/.jpeg) would resolve to glslViewer, whereas Image (.gif) would use VLC

Furthermore, I propose to write extensions more as bindings to an underlying application rather than displaying a certain type of artwork or file format. For example, as the video and image extension both use glslViewer I think they should then be merged into a single extension. Their code will be very similar, and the maintainer of that extension should easily be able to support all formats the underlying application can deal with. As another example, a VLC extension could display both videos and images and even Youtube URLs.

Decoupling the artwork format from the extension would make it easier to develop different clients in the future. A browser or Windows-based client, for example, might use other extensions compared to the Raspberry Pi client. These extensions would be catering for the opportunities or limitations their own environment brings.

Reuse extensions

I also recognised that a lot of extensions I would like to see on Openframe could be based on similar underlying applications. For example, a Twitter, Instagram, Text or even the .gif extension could be based on Chromium. It would be great if not every developer would separately have to maintain their strategies on how to open Chromium, deal with x server etc.

It would be great to extend/stack extensions instead. An example of how a Text (Twitter) artwork could be passed through three different extensions, each extending the functionality of the following one:

1. Text (Twitter) format would be opened by the Twitter extension which extracts text from a tweet and passes the text to Text extension
2. The Text extension would embed the text in a local web page, with some font and animation options and pass that web page to the website extension
3. The website extension would then start that local web page Chromium in fullscreen

---

Thanks a lot for reading.

What are your thoughts on this?

[jmwohl]

Interesting idea! On the whole, I like the idea of decoupling the artwork format from the extension. Thanks for thinking it through in such detail!

Taking your first section, I'm going to paraphrase the goal like this: As a frame owner, I want to decide what extension is used to play a specific artwork (or category of artworks, i.e. artworks of a certain 'format'). Is that fair?

So the idea is that Artworks still specify a format, e.g. Image (.gif) but not the extension with which to run it, correct?

Some more questions that come to mind: Who maintains the list of possible formats? Or is a 'format' just an arbitrary string? And if the frame doesn't have an extension that supports the artwork format, it errors?

WRT this:

Furthermore, I propose to write extensions more as bindings to an underlying application rather than displaying a certain type of artwork or file format.

I agree with this. One potential problem is that different formats will often require different commands or uses of the underlying application. E.g. your VLC extension might need different flags to properly display images, videos, etc. That means that the extension still needs to know all of the formats it can support, and have differences in order to support them. Maybe that's not a problem, since it's inherent in the architecture that an extension will define what 'formats' it supports.

SO, in terms of implementation, why store these frame settings as a prefs file rather than on the frame object in the database? There's actually already a formats field on the Frame model, but it could go in the settings as well. Something like:

"formats": {
    "Image (.jpg/.jpeg)": ["openframe-vlc", "openframe-glslviewer"],
    ...
}

That way everything is still manageable via the API without any confusion about what frame settings/prefs get stored on in the DB vs the filesystem. Is there any downside to this that I'm missing?

--

Layering / extending extensions is also interesting, but in order to keep the convo focused, let's address that separately.

[jvolker]

Sorry, for the delay in response.

Taking your first section, I'm going to paraphrase the goal like this: As a frame owner, I want to decide what extension is used to play a specific artwork (or category of artworks, i.e. artworks of a certain 'format'). Is that fair?

Yes, sounds good to me.

So the idea is that Artworks still specify a format, e.g. Image (.gif) but not the extension with which to run it, correct?

Yes.

Who maintains the list of possible formats? Or is a 'format' just an arbitrary string?

Maintainance is a good point. I had a semi-automatic process in mind. Each extension could define the formats it can handle. We could then have a crawler which would collect/fetch those formats from all available extensions on NPM. It could either be run manually or automatically on a regular basis.

When thinking about making the resulting list available to the user in the web app I could see this becoming overwhelming at some point. I think it would be good to bring the most important/used formats to the top of the list. Maybe we have a second manually maintained list with those most important formats. Also, I think we should keep the option to enter custom formats, not available in that list.

As an example, Processing is using a similar process to maintain a list of libraries which are listed from within their IDE/PDE. More about it here:

• https://discourse.processing.org/t/how-to-distribute-sketches-with-libraries-to-openframe/14095/2
• List: https://raw.githubusercontent.com/processing/processing-docs/master/contrib_generate/sources.conf
• Library example: https://github.com/atduskgreg/opencv-processing/releases/download/latest/opencv_processing.txt
• Crawler script: https://github.com/processing/processing-docs/tree/master/contrib_generate

Who maintains it? If we don't use a fully automated process I think it would have to be a maintainer/contributor to Openframe.

In addition, that list would be really useful in a web clipper browser extension to suggest artwork formats for a certain URL or file type.

And if the frame doesn't have an extension that supports the artwork format, it errors?

Yes, as it currently does as well. Ideally, the web app should give a warning as it already knows what extensions are installed on the frame.

E.g. your VLC extension might need different flags to properly display images, videos, etc. That means that the extension still needs to know all of the formats it can support, and have differences in order to support them. Maybe that's not a problem since it's inherent in the architecture that an extension will define what 'formats' it supports.

Exactly. The extension should define what formats it can handle and should throw a warning/error if it can't handle the format. It should also define what flags it needs and which ones are optional. The web app could then also display extension-specific artwork configuration.

That way everything is still manageable via the API without any confusion about what frame settings/prefs get stored on in the DB vs the filesystem. Is there any downside to this that I'm missing?

Sounds like a very good idea to me as longs as it gets implemented in the web app as well.

Let me know if I can help to implement it.