Rewrite libretime

[jooola]

The plan of bringing Libretime back from the old ages is still a main concern here. So I would like to propose a full rewrite of the project.

In order to do this I need to know a few things:

• First how much people are willing to contribute to the rewrite ?
  If I'm the only one and don't have any history with libretime, I don't see the point.

• How should we rewrite libretime ?
  I prefer to rewrite a minimal version of libretime in a separate repo and once the bare minimum is here, release an alpha version and invite people to migrate to the new version. Then implement all extra features one by one until beta, and finally stable.
  The other way to do this would be to start a new project within the old project and make a smooth migration, features by features. I'm a bit afraid of doing this since it would bring too much hassle to handle both new project and integration with the old one. Also this will not give use free hands for making big moves forward since we will not be able to accidentally break the state of master.

• Which technologies are the best suited for this project ?
  This question has already been asked here and there in this project, and it seems Python is the right way to go. Many tools are already written in Python, so we only need to polish them and ship them in the new project. Also the only parts of libretime that might endure high loads, is the streaming parts, so having a not so speedy language/framework is ok.
  Building a REST API has also been the global idea of a future project design. This is why I suggest using Django Rest Framework to build a well tested and self documented API for this project, and build a SPA for the frontend.

The main idea here is to use modern technologies to build a production ready software. Using modern paradigm like CI/CD or bots for all what can be automated (Avoid loosing time on things that shouldn't need much attention).

I already started to work on a Django Rest Framework project, and started to think at some design ideas/improvements, inspired by #2 (comment) but I need to know If I should go further, and if we can gather a nice team to do this work.

Also, since a client is willing to use this software in production, I might find some funding for this rewrite.

So I hope I said everything, probably not.

ping @Robbt @paddatrapper @gusaus @hairmare @libretime Who else ?

[gusaus]

Sounds like a great idea to me @jooola Curious to hear what @Robbt @paddatrapper and @hairmare think. Probably could be a good opportunity to create a goal on the LibreTime collective https://opencollective.com/libretime

[Robbt]

So I'm certainly interested in this but I'm going to focus on getting the current LibreTime alpha to beta and finishing work on some core functionality.

I think that decoupling the front-end from the back-end is a useful process. Currently there is a number of REST modules that could in theory be replaced by a python micro framework or Django or any other library.

We have talked about trying to do things with the StranglerFigApplication approach where you write new modules that tie into the existing system and then eventually replace it.

There are a lot of ways to do it, finding the time and resources to dedicate to it are the other problem. The initial LibreTime codebase when it was Airtime had multiple full-time developers being paid by SourceFabric to create the product. As a community working mostly in our free time we have been producing code but at a varied rate.

So yes, it would be great to rewrite Libretime, but the human labor required to do this is currently out of reach for most of the developers.

Personally I think that doing a lot of brainstorming in the pre-code level in terms of UI, API and the overall design of an ideal system would be useful. Looking at what makes LibreTime unique compared to other systems like AzuraCast. I have done a bit of the planning and analysis and would be willing to work on it but my free time is somewhat limited due to various other responsibilities in my life so I can't jump in and say yeah let's do this because I am not sure how I will find the time to do it in the immediate future.

In the long run we would benefit from having a more modern development framework to work from.

If we could define an API that handles the majority of operations that are currently performed via the web interface such as adding track X to show Y, Cancel show Z, Move track P to spot E. Create new show, etc. Then we could write the backend to accomplish said tasks and have mulitple front-ends.
This is tricker than it sounds but this seems like a reasonable approach to a rewrite.

In some ways the saas-dev port was already going about things this way as they started defining and using a REST api for tracks and podcasts but it was still dependent upon Zend 1. Podcasts was written using Angular 1 vs. datatables which the rest of LibreTime depends on. The trick is having enough momentum to actually move forward and making it worthwhile. These are some of my thoughts on the proposal.

[paddatrapper]

I agree with @Robbt, and the Strangler Fig approach is probably what I would recommend we do. If we define a solid API, implement it in some framework and then work on replacing parts of the UI as we the functionality is developed, we probably will have more success. This also ensures that people can upgrade their instances without much risk. This is something I think we could use the OpenCollective for, even if we are only providing a small amount to a developer each month.

From next month I can commit at least about 4 hours a week to this. I would us to plan out an API first before we start implementing anything. I see the following major areas of operation that LibreTime needs:

• Library - this is a list of songs/podcasts/playlists (both smart and not) that forms the base of a show.
• Item - a file on the disk that has metadata associated with it. This could be a music file, a podcast episode, or an advert source. These are uploaded to the server by users.
• Podcast - an RSS feed of files that are downloaded regularly to the disk.
• Playlist - a list of files that will be played sequentially.
• Smartblock - a set of conditions that match 0 or more items that will create a playlist at some later point.
• Show - a list of playlists and items that will be played at a specific time. These can be linked together to form repeating shows. A show can be connected to a live source which associates with a user's credentials.
• Advert Block - a point in time of one or more shows where 0 or more adverts can play.
• Advert Campaign - A list of advert sources that collectively have a start and end date and time. It also has an associated schedule of when which adverts should play. These are inserted into the relevant shows.
• User - an account that can log into the system. There are various permission levels that will grant the user access to different things.
• Live Source - a source of input that will be connected and streamed to.
• Destination - a streaming output from the system

Am I missing anything?

[paddatrapper]

This is how I see everything interacting:

[Robbt]

So the advert block and campaign are the only exclusively new elements here. I think that smartblocks should be reimagined as something like a collection of metadata that could be used as both a filter for the library and an selection for a playlist. Lots to do. I could probably also commit around 4 hours a week.

[paddatrapper]

So have a smartblock contain a list of rules that match things contained in the Library, so items, playlists or podcasts?

[Robbt]

Rather than focus on the specifics in this thread we should probably figure out a good way to build shared mental models for new design elements. I wrote up something as a comment here. I could elaborate more but I think it'd be helpful for us to discuss higher level. Like I said at the beginning my major focus is to continue working on getting our current system as solid and easy to install as we can with the existing paradigm while also contributing to the development of a next-gen redesign.

In terms of specific steps it would probably be helpful to conduct more user-focused design and really get feedback and support from existing users of the software, ie not just the developers or system installers but the people that actually use LibreTime to schedule shows etc so we can better focus our design on meeting their needs and not just making it easier to develop (although that is also a goal).

[paddatrapper]

I'm not sure how we are able to contact and interact with users outside of GitHub, which makes getting the design information from users tricky. I guess we could do some sort of poll/invitation of responses and publicize it on Twitter, OpenCollective, etc

[Robbt]

Yeah I think that we should spend some time designing a survey and then just promote it via various channels both online (top of README, discussion forum and social media) and informal (to people we know), it could include a place for people to add contact information if they are interested in getting further involved or being a part of a user interview. Just some best practices I picked up in my HCI coursework that I'd like to try to apply. It doesn't have to happen right away but it could also be a good way of engaging people and organizations that might be interested in helping fund the development.

[gusaus]

I think https://discourse.libretime.org/ is a good place to interact with users outside of github. Maybe we could pick up the recent discussion on development priorities and direct community feedback there?

Seems like the buckets of priority would be as follows?

• LibreTime 3.0 Proposal: Define our goals and roadblocks to release a 3.0 beta #544
• LibreTime API https://discourse.libretime.org/t/planning-out-an-api/341
• LibreTime rewrite Rewrite libretime #935

Chances are good LibreTime could be a pilot project for this Open Collective sponsored sustainer initiative once we're all clear on priorities and needs.

[gusaus]

I would like us to plan out an API first before we start implementing anything.

@paddatrapper Maybe we could start by referencing #935 (comment) in a new issue? Then we can incorporate relevant bits from https://discourse.libretime.org/t/planning-out-an-api/341 and other discussions.

[paddatrapper]

#1056

[paddatrapper]

Bit of a brain dump, but would like some input around potential pitfalls, easier options, etc

We are struggling with a fair bit of technical debt in our PHP and JS code. I think moving this all to a REST-based JS-powered front-end is a good idea. This is what motivated me to start the API Django code (#958). Ultimately this would mean that we could have our entire code base in JS and Python, dropping the PHP component we currently have. In the short term, I see us replacing views currently implemented in PHP with Vue-based views (or any other JS framework, I'm just familiar with Vue) calling the Django REST API. This would involve adding js/vue.js script elements to HTML pages and re-building functionality in JS. I'd suggest we keep the same look and feel, at least initially as we migrate over. After migration we can look at making things more mobile-friendly, reactive, insert-buzz-word-here.

I like this approach, as it means that we don't have to replace everything all at once. Rather we can do it a page at a time. Those pages that don't use the API, don't have the Vue script included. I'm not sure how well this plays with our PHP framework though, would it be possible to decouple pages from the PHP backend to it? How does Zend do routing and HTML templating?

[Robbt]

So I'm not 100% convinced that Django is the best option here or that we should migrate entirely to Python for our backend. I know from my limited research that Django is considered a batteries included framework and I'm wondering if something that was more orientated towards microservices might make more sense than it. There is also Flask and a bunch of other ones.

I am definitely in agreement that vue.js is a good idea, or at least I like that framework based upon when I've worked with it.

The biggest challenge I see is how do we maintain compatibility with Propel while moving away from it. Propel is evidently more or less abandoned based upon the failure to respond to the PRs I opened to fix compatibility with the latest version of PHP and it is also the interface with Postgres. Would we basically have to roll our own ORM and write all of the queries to maintain compatibility with the datastructure that Propel expects.

You've done more work on this, so I'm not trying to say we should discard the Django effort but I am just curious as to whether other options were considered and/or if it is the best option. I have never taken the time to learn Django but I do like the project. There is also laravel in and the community driven fork of Zend the laminas project.

Yeah I'm not sure about the best way to approach decoupling and running in parallel but there is some guides to how to do that for Zend 1 to Zend 2 migration because basically you had to rewrite everything to upgrade. https://framework.zend.com/manual/2.4/en/migration/overview.html

Another thing is that LibreTime is essentially written on datatables.js basically the Library and everything is dependent upon that javascript library and it is jquery based and a major pain, but also pretty fully featured and well we have our own hacked version of it. If we were to replace that it would be huge but it also essentially would require replacing the entire UI. We never made much progress with making LibreTime work without the hacked components so that newer versions of the libraries can work.

Thanks for putting the time into thinking about this. I'll continue to contribute as I can.

[paddatrapper]

I suggest Django largely because I have used it a lot, so I am very familiar with how it works. Flask is the other big framework, but you have to do essentially everything except map HTTP requests to python functions. So all database and REST logic would need to be implemented by us. In comparison #958 has a fully working DB-backed REST API (with open access to everything) in what amounts to 2 files and a diff of 1 239 changes.

I've never used Laravel and other server-side scripting languages are definitely a possibility. I like the idea of having all our code in the same language though (daemons and HTTP backend).

The way I have done it so far is let Propel handle the DB and all migrations that are required. The Django server maintains the same schema (in api/models.py) and does the translation to Python classes that can be interacted with like normal. This is a very neat Django feature for dealing with legacy databases. Once we decide that we no longer need the Propel ORM, we switch over to Django handling migrations (changing 1 line in each class and making the initial migration). This would require people to upgrade in two stages if they are jumping from releases. E.g. say 4.0 includes Django ORM, they would need to upgrade to 3.9 if they are on an earlier release first and then 4.0.

https://framework.zend.com/manual/2.4/en/migration/zf1_zf2_parallel.html#run-zf1-and-zf2-together suggests using HTML server URL rewriting to manage sending between ZF1 and ZF2. This could potentially work too. I'd like to play around with seeing if I can get a Vue page to play nicely with our current frontend...

I'm expecting we would need to re-write the UI anyway to get away from things like datatables, etc, but we may be able to get away from the PHP stuff first and then look at replacing the front-end look and JS libraries afterwards. Alternatively we live with two different page look-and-feels while we migrate - so feature freeze the current UI and replace it page-by-page with a new one with Vue/some other JS framework and as new UI features are implemented, require that they are implemented in the new frontend. The old pages would still look for they do, but would be phased out slowly

[gusaus]

Just on the surface, it seems like one benefit of Django is more folks like @paddatrapper and @jooola #935 (comment) are familiar and could potentially contribute...

...once #544 & #1056 are out the door 🚀

[jooola]

I asked once if golang could be the right language for the future of libretime. If the micro service architecture is the way to go, I think golang might be a really good choice.

Someone might also said that, but micro-service architecture could also provide an incremental approach for this big work. Additionally, a micro service architecture gives us the possibility to mix some programming languages.

I'm just throwing this in the discussion, maybe some will share this idea.

I admit I am more familiar with Golang than Django, so this should bring me closer to such a project. And if a micro service architecture is chosen, django doesn't seem to be the right tool for it, still needs some more investigation.

[kmahelona]

I've also got a lot of experience with Django and Django Rest Framework (DRF). We're now building apps with DRF as the API back-end and vue.js as the front end (sounds like we're sold on vue 😉). We also use FastAPI (python) for performance on some of our Machine Learning work. Our choice of python is because more of us know it and you can use it across most of the stack. I'm not saying these are the best options, they're probably not, but it's what we're using and it works well for our context.

I couldn't say if Django is the right choice here, although I would have chosen it if I had to build LibreTime from scratch. One thing I like about Django is how it handles migrations, but we have been using postgres JSON fields more regularly which really means you kinda don't have to worry about migrations unless of course your front end is heavily dependent on that JSON field -- then you're back to manual migrations. Also Django supports web sockets now which might be useful for LibreTime. I can imagine in the future... we might want to build a browser based way for an announcer/DJ to go live (webrtc audio streaming to liquidsoap) and something like web sockets might be more appropriate for that sort of live interaction 🤷🏽‍♂️.

I do think the micro-services approach makes sense. LibreTime is kinda a bunch of micro services at the moment. Even in our stacks I'd still probably use Django as a "micro service" for certain things because it's super fast and easy to do stuff, but then maybe the "core" is run on something more appropriate.

[zklosko]

What does everyone think about moving to an API-driven database, like CouchDB or MongoDB? Pulling JSON directly from queries could minimize dependencies and make the rewrite easier on us.

Personally, I'm voting for CouchDB because it's easier to install on my computer for dev purposes. 👍

[paddatrapper]

I'm not sure a schema-less DB would be the solution, as our data fits a relational database well. Django has migration management built in, so we can use that once #958 is finished and merged.

[stale]

This issue has been automatically marked as stale because it has not had activity in the last 5 months. It will be closed if no activity occurs in the next month.
Please chat to us on discourse or ask for help on our chat if you have any questions or need further support with getting this issue resolved.
You may also label an issue as pinned if you would like to make sure that it does not get closed by this bot.

[paddatrapper]

We now have a Django driven API and a WIP Vue frontend (#1122). So I think as an over-all proposal, this has been completed. Now the issue is implementation