Web Interface Stack Upgrade

[sbalbrecht]

Following up from the beginning of this discussion. I am new to SABnzbd and open source contributing but I saw a need here and feel motivated to make some contributions.

The technologies behind SABnzbd’s UI haven't been updated in some years, with some being leftover from even earlier refactors. Many of the libraries used are old and/or unmaintained, and some obsolete. This discussion is to analyze and understand how the current stack is used and form a plan to migrate to more modern and maintainable alternatives.

The goal is for the change to be transparent to the end-user. Changing the look and feel of the UI is not part of this upgrade plan, but could be a separately-planned project.

I started writing a section on potential UI frameworks but there is a lot to unpack, so I thought it would be wise to split that into a separate discussion after establishing a high-level plan.

Current Stack

These are the libraries involved in the UI from what I've seen poking around the project.

• Cheetah
  • All HTML templates are defined with Cheetah and its control flow is heavily used in templates to generate the UI
  • It is an old, small project, so it would be desirable to migrate to a modern UI framework (e.g., Angular, React, Vue, Svelte).
  • Still poking around, does it use server-side rendering or are the files statically generated?
• jQuery
  • JavaScript is now capable of doing everything jQuery does, albeit less concisely.
  • Used heavily in the project for conveniently selecting elements and maybe making AJAX requests.
• Knockout.js
  • Used to bind Javascript variables to the HTML, and update the UI when the state of the JS changes. MVVM pattern.
  • Hadn’t heard of this library before, but its functionality is a basic feature of any modern UI framework
• moment.js
  • Super useful in its day, but is now in maintenance mode. It is a heavy dependency for what it provides.
  • The project itself recommends switching to more modern libraries like Luxon or day.js (much lighter and actively maintained)
• Bootstrap CSS components
  • The bootstrap look and feel has been heavily modified so it seems mainly used as utility and convenience (quickly styling buttons, a working navbar, organizing elements in a grid).
  • The version of Bootstrap in the project still requires jQuery, while new versions do not. If we needed to stick with Bootstrap we could still eliminate jQuery.
  • CSS has improved a lot since Bootstrap’s inception. CSS grid and a load of other convenience features make a lot of Bootstrap’s utility less valuable.
• Any others I’m missing?

Potential Migrations

1. jQuery can likely be eliminated without insane effort. jQuery is used heavily but the migration steps to native JS are fairly straightforward. Eliminating jQuery (besides its required use in Bootstrap) would probably be a good first step.
2. Moment.js does not appear to be used for too much. I haven’t looked into how the locales would migrate to another library like Luxon/day.js but I would be surprised if this was a big issue. Eliminating moment would be a good second step, but more research would have to be done to decide on the best replacement. I haven't researched Luxon but I know Day.js is tiny, designed to have a similar API to moment.
3. Conceptually I am lumping together a migration from both Cheetah and Knockout.js. The functionality of both projects should translate easily to a modern UI framework, both in templating with reusable components and binding and updating the UI in response to changes in state. We can review possible candidates separate from this discussion. This would be a good third step.
4. We could migrate from Bootstrap to either native CSS or something like Tailwind, or simply upgrade Bootstrap to the latest version. It depends on how and why it is used in the project, so I’d like more context on that. With modern UI frameworks making it easy to build isolated components, I think the meta has shifted more heavily in favor of Tailwind’s approach. The main opposition I’ve seen to tailwind is that it can bloat the HTML with many classes, but I’ve found with components this is less of a concern. Design philosophy behind Tailwind.

[Safihre]

Good analysis and I'd like to jump in straight away on your suggestions for potential migrations:
As you are a software developer you probably agree with me (I'm a product manger at a software company) that we should try to keep the stories/changes as small as possible and try to prevent waterfall deliveries.
The UI stack for Glitter and the Config are slightly different, but they do share a few common components like Cheetah (server side template rendering) and the API. We should probably seperate changes to the Config and Glitter to keep things as light as possible.
Your first suggestion is jQuery, but I can already tell you that unfortunately it's used much more than appears from first glance. There's of course the Bootstrap usage, but we also use jQuery-UI for the drag-and-drop, and several plugfins like typeahead.
So, I would not start with that one, I would suggest the following:

1. Replacing Cheetah by modern template alternative. This way you can already see large parts of the UI code, identify parts that can be replaced by API-calls and others that should be replaced by API calls instead of template-based logic.
2. Update the API to use a modern engine to add new API-calls in a more modern way. @jcfp or I can do that.
3. Update Bootstrap and jQuery.
4. Update UI-library.
   The last ones are deliberately unspecific, as we will have to see how well the first steps go 😁

What do you think?

[Safihre]

Awesome. That po file looks the same like ours, right?

[sbalbrecht]

Yep!

[thezoggy]

is there any good way to measure the impact/benefit of the front end stack change in a meaningful way?

with sab paused (so we focus on front end and not adding the backend downloading stuff), refreshing the front page.
how much memory used / how long it took to load / cpu usage... when sab is idle with empty queue vs lets say 10k items in the queue.

i know things would be imperfect but ballpark to know hey we know the change is going to yield net benefits to warrant the change.

[sbalbrecht]

Yeah, we can measure bundle sizes for each page load before and after, both initial page load and load when navigating, and initial rendering time of the content. I expect the SPA to be snappier and lighter during navigation, and the initial bundle might even be smaller too. I can take some of those measurements when I'm a bit further along, but yeah there are a bunch of metrics to compare.

[Safihre]

@thezoggy but I doubt the load on the server side will be lower. I already squeezed out as much as I could from the main API calls.
Replacing Cherrypy will help I think, but that's a good one to measure for huge queues.

[sbalbrecht]

Hey all, checking in with a hefty update.

Following up from last week, I figured out translation with LinguiJS and now translation in the UI is as simple as using the <Trans> tag or t macro using the English string directly, which I personally find much more readable.

<Trans>SABnzbd Quick-Start Wizard</Trans>

<input title={t`Select only if your provider allows SSL connections.`}
    ... other attributes ...
/>

I spent this week migrating /wizard/one. It was a good stepping stone as it required me to get familiar with libraries for form state management, routing, and data fetching. I also migrated the imperative state management logic to React’s declarative approach, which was satisfying. The only jQuery I haven’t yet migrated on the page are the tooltips.

There are a few improvements to make regarding routing and data fetching on each page load due to my inexperience with the libraries and patterns, but overall I think this is a good foundation for how most functionality can be migrated.

However, now that I have a better idea of what the migration will look like, and before I go much farther, I want to raise some concerns about the decision to introduce a node-based frontend framework to the project.

I recently reread https://grugbrain.dev/ (among other essays) and its warnings of the dreaded complexity demon. The section on front-end development struck a chord:

"I know, I'll split my front end and back end codebase up and use a hot new SPA library talking to a GraphQL JSON API back end over HTTP (which is funny because I'm not transferring hypertext)"

now you have two complexity demon spirit lairs

and, what is worse, front end complexity demon spirit even more powerful and have deep spiritual hold on entire front end industry as far as grug can tell

The existing UI is kinda fragmented with different outdated libraries, but the stack is super simple with Python serving the Cheetah templates.

With React introduced, there are already a lot of moving pieces:

• node.js and npm as a separate runtime during development and compilation
• React as the core framework for state management and UI componentization
• Vite for the dev server and bundler
• LinguiJS for localization
• @tanstack/query for data fetching
• @tanstack/router for routing (could instead go with react-router, but I liked the type safety and unification with query)
• react-hook-form for form state management
• zod for form schema validation to be used with react-hook-form
• Bootstrap still remains, but could be replaced with something like tailwindcss

Even though I haven’t yet explored the requirements of Glitter and the Config, I don’t anticipate too many more big libs for the frontend, but the point stands.

After some research, it seems like Cheetah, Knockout.js and jQuery can be replaced with more modern alternatives without devoting ourselves to the heaviness of a JavaScript frontend framework.

For example:

• Starlette docs recommend Jinja for templating
• Embracing templating over JavaScript, Knockout.js and jQuery could be replaced with htmx for HATEOAS, using something like alpinejs to supplement the scripty-bits that are hard to implement using hypertext alone.
• Bootstrap stays for now but could be upgraded to the latest version that drops jQuery.

While this is a less conventional stack given the state of frontend today, it is simple (and I mean htmx was 2nd in the 2023 JavaScript Rising Stars "Front-end Frameworks" category, #10 overall).

You get to keep the benefits and speed of SSR, have relatively few dependencies, and keep everything within the Python backend instead of splitting the frontend.

I get the draw to use a JavaScript SPA framework because they are the predominant solution today and there are plenty of resources, but that decision adds a lot of complexity to a previously Python-exclusive project. As we have established that there are few contributors to SABnzbd, so the most important thing is that the stack is easy to digest for existing maintainers. Making the UI in React is doable, but maybe more complicated than SABnzbd really requires.

Let me know your thoughts. I would be willing to start a new branch to see what the Jinja/htmx/alpine stack might look like.

[sbalbrecht]

I will take a closer look at the Glitter js, but I do think all the interactivity is achievable with vanilla js and htmx/alpine with some ingenuity. We can eliminate knockout and jquery without moving to a js SPA framework. I'll give an update in a week with some thoughts.

[sbalbrecht]

I picked this up last year, been a pretty good read:

https://livebook.manning.com/book/microservice-apis-in-python/

Also, I will take a look!

[sbalbrecht]

Hi all. Life got busy for me over the last two weeks and isn't stopping, so I haven't looked into this migration directly since my last update.

However, I had the opportunity to rewrite a UI at work using thymeleaf (Java templating engine) and htmx and learned a lot.

I was able to implement infinite scrolling, polling server status, replacing table rows, etc., without any JavaScript. The only thing I used JS for was for sorting table columns, but this was a preexisting solution that I didn't have the time to migrate -- still doable.

With this method of development, instead of receiving JSON from the server and JS updating the UI based on its state, you receive a block of HTML with which you replace a portion of the UI, without refreshing the entire page.

For example, if you want to edit a table row, you click the "Edit" button on that row, which sends an HTTP GET, and the response is that same row with inputs inside the table cells. After editing you click "Save", which sends an HTTP PUT, and the response is the updated row with static content. The entire state of the UI is communicated with hypermedia alone, without JS.

This means that most of the logic for configuring the UI according to the state of the data is handled by the templating engine at render time, using endpoints to handle changes in UI state instead of handling changes with client-side code.

You are potentially creating more back and forth between client and server to handle UI changes, but you are also aren't shipping JS bundles which would normally be used to do that work. Since most apps already contact the server to communicate state updates with JSON, this seems to balance out.

Another consideration is that HTML is a heavier format that JSON, but again, the JSON requires JS to interpret so the UI can be updated, so it balances out in a similar way.

Also worth noting htmx recommends splitting your application API from your general purpose data API as opposed to using content negotiation in the same endpoint, mainly because the general API can be kept stable while the app API changes with features (and, only server hypermedia).

So I have found communicating state with hypermedia works well for a lot of seemingly complex interactions, but becomes more difficult to work with if the app has many dynamic components, i.e., you will have a bad time implementing Google Sheets using hypermedia alone.

Glancing through the Glitter JS, I suspect a lot of the Knockout functionality can be migrated to this approach, but y'all would know better than me about tricky parts. I still plan on trying it out when I have more time.

[Safihre]

You describe exactly how SABnzbd used to work 10 years ago 😅😅 all the skins worked exactly like that, using the template logic.
It was an utter mess...

[sbalbrecht]

Oof!

I sense you do not want to go back down that route. In the meantime, migrating from Cheetah to Jinja shouldn't be too bad.