Multiple Feature Requests

[convivitybackendteam]

Hey Elonen,

I hope this message finds you well. I wanted to reach out regarding your project. I think clapshot is a great addition to the open-source and I believe a lot of people will benefit from it.

I have a few suggestions for features that I believe would further enhance the project. I would love to see the following added in the future:

A login system to secure user data and content
An admin panel with roles and permissions to manage users and content effectively
The ability to assign folders to specific users for efficient content management
Workflow automation to streamline processes and improve productivity
Thank you again for your dedication and hard work. I am excited to see what the future holds for this project.

Thank you!

[elonen]

Thank you for reaching out! I moved this from Issues to Discussions as there were multiple high-level ideas here.

Authentication / login system

I've added a new HTTP basic auth Docker demo today. It shows how to do simple user management with an external web app ("htadmin"). It might even be enough for some small real-world deployments. See https://github.com/elonen/clapshot#multi-user-demo-with-http-basic-authentication for details.

Modern IAM is such a complex topic that trying to add built-in user management would almost certainly result in security holes, tons of extra research to do, code to maintain and requests for endless auth method variations. In fact, auth inflexibility of some existing alternatives was one of the main motivations to start this project in the first place! That's why I made a decision to keep login / user management stuff out of Clapshot itself as much as possible, having admins do it on in Nginx (HTTP/Websocket level) according to their own policies and protocols.

I know it makes casual deployment a lot harder, but it also has allowed me to concentrate much more on core features.

Authorization granularity / roles and permissions

As you correctly point out, Clapshot currently doesn't have any notion of projects, folders, labels, tags, private or public files etc. It works ok(ish) without them in the ongoing project I mostly designed it for, so other features are more important for now.

There have been some requests about this from other users, too, but I haven't yet formed a clear picture on how it exactly this organization/authorization stuff should ideally be designed to strike a balance between functional generalization and usability.

Granular access control needs to also be easily integrated with LDAP ACL groups and other external authorization schemes. In addition to taking care of auth, Nginx also currently streams video files directly to the browser without Clapshot API server acting as an HTTP relay, and I'd really like to keep it that way.

These points mean it would probably have to map project/folder/label names to URI paths in some way, so as to make authz lookups possible on Nginx level. How to do it best is not quite obvious to me yet, so I've been waiting for more user experiences and discussions before making any architectural decisions.

Workflow automation

Could you give some examples on what exactly you'd like to see here?

Some kind of scripting framework should be doable, but the design philosophy will have to be much more decoupled and low level than in large commercial products such as Ftrack. I'm pretty wary about complicating the codebase so it doesn't become too unwieldy. Descriptions of real-world priority use cases would help to figure out a minimalistic interface to make them possible.

[elonen]

Writing down an idea about combining granular authz and workflow automation stuff:

Frontend could specify a set of UI widgets/operations like folders, check boxes, text input fields etc. that the backend could then use for organizing video listings and offering custom metadata fields - both editable and non-editable.

The module generating these at the backend, let's call it Organizer, could be a plugin, perhaps user-editable embedded Python. It could then, for example, lookup configured projects and their participants from LDAP, SQL or whatever the user needs, and display highly customized workflow information, specific to their organization and drawing from other data sources like project management databases, Google sheets or what have you.

Nginx would be configured to make an auth subrequests against Organizer with URI and username, and it would then determine authorization to the video (playback and commenting separately, maybe) based on anything the user wants: project membership from AD groups, expiration date (e.g. temporary links), user's work title, workflow state, video title, uploader's name etc. This way, video URLs wouldn't have to encode information beyond what they currently do - video hash.

Default/example Organizer script could perhaps use another .sqlite file to implement user specific folders, private videos and projects (maybe tags so a video can belong to multiple projects..?). Hints about how to adapt it for LDAP in code comments.

[convivitybackendteam]

Hello Elonen, I appreciate you taking the time to reply.

Workflow example:
We want to start a workflow when a user uploads a file. It has a different kanban board style layout and shows different stages of the content cycle. Example: Video recorded, Video produced, In review, Re-record and done. On change of status, can we send the video data using webhook, For example when a video is moved to the 'Done' stage, can we upload the video to our LMS and into a lesson?

This can be achieved with n8n and some other tools. But we need some sought of integration.

Login and Roles
A login can be added to the project by using open-source login and auth systems like supertoken, ory, keycloak, etc some of them also have roles and permissions.

Folder
What are your thoughts on assigning folders to users?

We really like air.inc and frame but they lag in some features. So, we can't use them.

We are online ed tech and we want to make our workflow easy and automated.

Thank you so much for such great work!!

[elonen]

Thank you, these concrete examples were really useful.
My local branch now has two new features that I'm planning to use to tackle both workflows and folders:

• Embedded Python runtime
• Simple "props" table in database (key->value, possibly json)

The current idea is that when a user opens their main page, Clapshot server would invoke Python, that would in turn then return a JSON that tells the frontend how it should render the video listing. In the default/example script I'll probably just lookup user's private folder tree from the new props table, and show the videos inside them at the frontend. The grand idea though, is that you can then customize that script to show virtual folders (perhaps the same video in multiple folders), your own popup menu entries, video tags etc. And of course, add similar script hooks to other events such as video upload, transcode finish and delete.

Your custom kanban style workflow could then be visualized either by showing a virtual folder per process stage, adding a tag to the video UI element or something else, depending on your needs. Your Python script could pull this workflow information (along with user's workflow stage transition privileges) either from Clapshot's props-table, or some external source like your own project SQL database, LDAP etc. Or perhaps calculate it on the fly based on some project-dependent criteria.

Do you think this architecture would work in your use case?
It's very general in purpose, as I've received similar, but not identical enough, metadata / content organization requests from other users.

Next I'll need to implement folders and other UX elements on the frontend. There are quite a few different ways to make them, so if you have preferences, now would be the perfect time to draw a UI mockup picture or two. :)

[convivitybackendteam]

Folder design: https://www.syncfusion.com/javascript-ui-controls/js-file-manager

I am sure you kept a way that the admin can see all folders and is there a way we can integrate s3 or any storage system?

I am happy to give you a junior developer resource (he is still learning), let me know if that will help you in any way.

Would you be interested in a call to explore possible collaboration? I have some ideas which can help you develop the platform faster and it will be a win for everyone.

How does that sound?

Take care!

[elonen]

Thanks for the design reference. I have a pretty well working frontend implementation draft that actually mostly follows that logic. Some bugs to iron out, but nothing major. Folder tree on the left in your reference looks like a useful improvement. Maybe not a first priority but at some point. Here's a work-in progress screenshot of the folder UI:

From a technical high perspective, there are three parallel development branches in the project currently, that I'll merge for the next release:

• client-dnd -- Refactoring, draggable folder structure support for the frontend (screenshot is from this), and Organizer plugin -controlled popup menus for custom workflow actions

• plugin-grpc -- Organizer plugin support for backend. I ended up ditching the embedded Python direction, in favor of gRPC, which allows the integrations to use almost any language you want and still provides a type-safe interface. I'm writing the example/default plugin in Rust for now, but might port it to Python as an example later. Even though gRPC is remote protocol at heart, my current design supports/recommends that you give Clapshot server a shell command that it then uses to run the plugin as a subprocess and connect to it over a Unix socket file instead of TCP, for maximum stability, performance and integrated logging (Clapshot server logs whatever the plugin prints in STDOUT/STDERR).

• props-graph -- Simple property graph implementation in the same database, using new prop_nodes and prop_edges tables. Organizer plugins can of course access any data sources they like, but this simple integrated graph db provides an easy way to add some lightweight metadata while maintaining a decent level of referential integrity in the .sqlite file. See commit a200a05 for current schema.

These are all parts of a larger effort to implement a customizable folder UI:

1. props-graph will provide a simple graph DB that
2. plugin-grpc (example Organizer plugin) uses to construct a (drag-and-drop re-orderable) tree hierarchy, that
3. client-dnd then displays on user's browser.

I'm currently doing this mostly as an unpaid hobby/learning project, but partially also as a paid effort to support a commercial project that needed a self-hosted tool like this. While the current functionality is mostly sufficient for the said commercial project, users there have - unsurprisingly - asked for similar UX and workflow enhancements as you. The paid work-hours I can put into their requests are limited however, so my hope is that Clapshot gets improvements from other contributors like you, and I can then provide them with a better tool than I could make by myself alone. To that end, it sounds great that you have some development resources and ideas you'd be willing to spend on this!

While I'm working towards a release-ready version of the Organizer plugin interface, I'd suggest you start preliminary R&D for your workflow integration by making a simple gRPC server in the language of your choice, and planning exactly how you'd like to hook different Clapshot events to your workflow.
I'd be really interested in hearing about those plans, as they'd most likely help me make the plugin API more useful for practical cases I haven't thought of yet but would encounter soon enough with my own users anyway.

If you want more gritty technical details right now, drop me an email and I can send you some of my preliminary designs for how I'm planning to implement the basic project/folder Organizer with the components I described above. It might not be super efficient use of time at this point though, as the upcoming release with an actually working default Organizer will make things a lot more concrete than reading through my pseudocode plans.