Modernization of source code

[skewty]

As a full-stack developer myself I was excited to learn about this project as I felt it would not only be of use to me at home but that I could also give back via bug fixing, implementing feature requests and perhaps tackling some new features.

After a quick git clone I was a bit surprised to see:

• no type hints in server side Python source code
• old libraries used in client side JavaScript source code

While users may not care what the back-end source code looks like, contributors certainly do.

Python with type hints (first available in 3.5) and React + TypeScript, IMO, are the way to go to lower the time it will take for contributors.

While React may not be my favorite, it is easily the most popular and thus more people can quickly understand the code and contribute. IDE support for REACT and TypeScript are now top notch. JavaScript 1st libraries (with @types typing or types specified in comments) typically offer spotty type hinting at best and often rely on duck type magic / value inspection to decide how to treat parameters. This means IDE's have no chance of helping you.

I myself wanted to add image support to the Rock-ons screen (after seeing some JSON file have them specified already) but quickly got lost as I wasn't able to follow certain definitions (magic global values) from where I was into unknown other files.

The entry level understanding to contribute to this project is so much greater when types must be guessed (python) and on the client side, a very legacy "backbone magic" naming / structure approach is used. One can't for example "goto definition" or "check for usage" using a powerful IDE (JetBrains) because of the way source code is written.

So I am wondering if the core team has discussed any of this and what were their thoughts / decisions. Is there a discussion or a place to start this discussion?

[phillxnet]

@skewty In the base proposal I am agreed. But all this is far easier said than done. Pull requests welcome and all that. We have just underdone a major uplift on another sometimes overloooked element from the pure development side of our base OS. We went from many years being CentOS based to, to now being "Built on openSUSE" this in itself was a major undertaking and took pretty much all our developer resources. But we at least now are based on an OS that supports our base filesystem.

Our next major focus is on exactly your first point as it goes:

• no type hints in server side Python source code

Or more specifically it's overarching requirement:

Python with type hints (first available in 3.5)

See issue: move to python 3 #1877
which in turn has mean addressing an entire cascade of dependencies, the first major one having already been 'sorted' via:
Replace orphaned django-ztask with Huey #2276
which is now part of our stable (master) branch and released as such in our new installer as it goes.
The next major 'stage' of development under the umbrella of Python 2 -> 3 is our base technology base of Django:
Update Django within current Python 2 constrains #2254 #2365
Again already merged and awaiting it's sub-step of returning all major function before we can release this as the beginning of our next testing channel. We are now based on 2 branches (master and testing) by necessity to address these massive changes. See:
"As from 4.1.0-0 stable release, the new testing branch is default for contributions." rockstor/rockstor-doc#357
And this ongoing 'return to mostly functional' testing branch work is covered in the ongoing:
(t) Failing unit tests #2371
which aims to return our hundreds of unit tests back to a healthy response/state so that we might be at least nodding towards good development practice and honouring our users with some degree of proven functionality.

Several of the above issues are referenced in the meta issue of:
WARNING -- EARLY TESTING CHANNEL -- KNOWN BROKEN (DEVELOPERS ONLY) #2362
Which will feature in the first testing channel rpm release changelog to warn folks of our massive change of having a known non working code base. We have for almost our entire development history had working testing channel rpms but the enormity of even just changing our internal scheduler (to Huey) and then our entire Django over several major versions in order to even consider our Python 2 to 3 change you initially indicate just so we can have nice things (read Python 3) is, as stated, easier said than done :).

An alternative approach is to upgrade everything and hope it all works. We have that as well actually! Via the heroic and likely overly optimistic but much appreciated efforts of @Hooverdan96 via:
(t) Experimental Django update #2372
Which is not, as far as I know, functional in any way or can even build. But I'm not throwing away the likely gems that lay within this work that we are likely to need in the future (Note the full on approach of all things updated).

We also have, known to the core team already, a build dependency issue in that key elements of our build system are essentially unmaintained. See Buildout, which has yet to move itself to Python 3 and we have also setuptools that buildout continues to use for purposes that it itself no longer supports in it's later versions.

Green field is easy. One picks the latest and smashes them together and they are all aware of each other. Our ztaskd was looks perfect at the time. Several releases later it died a death, but even given this it took us several years into our future at that time. But ultimately we were forced to replace this hard dependency due to the lack of Python 3 compatibility (oh and the complete lack of a ztaskd maintainer for many years of course). Likewise 'back-then' buildout was all the rage. But we would now like to move to poetry for instance and dump the entirely of builout. It just holds us back now. But before that we have a working build system and we need field testing via our testing channel and for that we need all our tests to pass; ergo:
(t) Failing unit tests #2371

So the key here, I believe, is to develop in manageable and maintainable steps where we can prove function from one major change to another. We still have some threading issues form our introduction of Huey, but if we can fix the testing to correct the likely missing pieces of our Django move to a version (still old granted) that can carry us over the critical pivot point of Python 2 to 3 then we are on the home run. But in the interim we have folks who help to support our efforts that expect, rightly, that they can easily test and contribute. Which again is a chicken and egg. Especially if we move everything over to brand new and shiny only to have nothing at all that builds.

So in short we are on the case and need help, not necessarily advise of the type "this could be re-written in *" I.e. Rust for example, or as is contextual here Python 3. Agreed actually. I'd like to have web assembly as our base for the Web-UI, but that is the purpose of Django which is now one of the most well established Python Web-UI frameworks out there. Sure we could do a jump (down) to Flask, or the by all accounts excellent fastAPI, whos author incidentally called Django an extremely elegant project. But we are now years into Django and have many Djangoisms which as seasoned developers will will often involve complete re-writes to write around. We have few active developers and I fully agree that we have a typical old mess of dependencies. But I see the best way forward as to update that which hosts our users data (done via CentOS to openSUSE) and that move in turn turned up some way deep works-by-accident elements that took quite the development time to fix. It also involved abstracting our systems manipulation (read OS) so that we dealt with both CentOS and openSUSE for a couple of years actually. But then older dependencies (setuptools as it goes) caused a fatal build issue in CentOS that we simply didn't have in our newer base of openSUSE. So in short we are in a technical dept situation and I think we need to carry on regardless with the pan as laid out in the above meta and more focusted issues. As we can then have the nice things that you reference in your initial "no type hints in server side Python source code" which are simply, as you state, non existent in our current base. So we update our base, and the now merged major change in Django should in turn lead to this. But we must, as much as we are able, maintain buildability and runnability. Ideally with some proven function so it eases user ability to feedback. We have many many many more users than developers. So as usual the developers have to soak up the graft of the technical dept. But that is, as indicated, under way. But we also have to pay for our servers and that has recently deleyed at least my effrots on the indicated efforts.

I'll not comment on your preferred REACT and just way that our Django in and of itself dictates many of our technologies. And we have had folks attempt, unsuccessfully actual, to up some of our base front end technologies. See for example:
backbone: #1484
Reason, to enable intergration with what was not native at the time but is now snapper; outcome complete Web-UI dysfunction.

As you obviously appreciate within your statement:

While users may not care what the back-end source code looks like,

Functionality is king. The best looking code in the world is of no use if it doesn't work. But to be a little more fair on us our Python code is system wide Black formatted: established via the merged long ago by the venerable @FroggyFlox:
Project-wide Black formatting. Fixes #2195. #2199

Large projects, like large boats, have to move slowly if they are not to sink. Try dragging/pushing a oil tanker at typical speeds associated with far smaller boats! Example (sorry @Hooverdan96 but it's a perfect match and we have already discussed this to our mutual benefit I believe) the above pr to change all dependencies at once: what builds? What works?

I'm delighted that you are interested in the workings. Few enough are 1 interested and 2 able (the first being the only prerequisite given we have many first time coders/contributors in our pull request record). Point 2 can almost always follow with interest.

Jumping into a project that is pushing a decade old and saying lets re-write in another language is often not the place to start. And we have already moved our OS, which in some ways is more difficult. Many Web developers who claim 'full stack' are not. They know nothing of the underlying OS. That is our meat and potatoes. Ultimately if we don't play nice with not just a web server but with an entire OS, we are toast and worthless to anyone. We must play with the entire OS and on that front the Python 2 to 3 is critical as Leap 15.4 is now out and has no Python 2. That is an issue that is more pressing than if we should be REACT based. Which incidentally we will never be. Talk about adding complexity, or completely re-designing ;). Many early REACT projects have now dumped this in favour of naive tools. We are entirely and completely happy with Django but yes, we would very much like to have channels 2 available, and be using newer async throught-out. But first things first, we need to move our Python, and for that we need to move our Django (done almost) and to prove our function and resource our most valuable nature (open source) our users. Thus the testing issue featured above. Once this is done we release a new testing channel rpm and do hand-in-hand development with the community.

Developers always welcome but it is rarely helpful to suggest your favourite framework of the day as a complete re--write.

Again documentation is always king on this front and yes we need to update this. But to be fair I recently completely re-wrote this to account for our OS change. Few non appliance apps need worry outside even their pip prescense in our Python reality. We need to also cover our OS install. See:
https://github.com/rockstor/rockstor-installer (9 contributors to date not including spin off repos)
Also completely re-written (technology wise incidentally) from our prior:
https://github.com/rockstor/rockstor-iso (2 contributors in may more years of 'life').

So yes I thing we understand the need to change when needed. But our preference in Python and Django and whatever the mirriade of old stuff we currently have bolted on for the UI. But note that Django doesn't yet support web assembly as I for one would love to dump the abomination that is JS and any and all of it's derivatives (build on quick sand and you fail). But I try to stay pragmatic for the good of the project as a whole. So for now I haven't ripped out our JS components ;).

My apologies if this post comes over as unfair. But you have not fully appreciated the depth of this project I suspect. Full stack is to the metal, not to the web server. We needed first to give our users a new OS where btrfs was not just a technical preview that was about to loose even that status (see CentOS) and then be followed by a complete renage on CentOS itself. Hence our openSUSE move which I'm personally chuffed with. We can now move forward with our 'other' technical dept. We did as many projects do, move faster than we were able to reach a working state, then backtracked to address all that had changed in the interim. In our case we and many projects will always be doing this. But complete tec switch-out or language changes are almost never needed. But our Python 2 to Python 3 was forced on us. Python 2 was just too loose and that has now self corrected and we have the famous dogs dinner of 2 -> 3. An element of our Django issues and our buildout issue and every other python library in existence. If I could have chosen Python 3 form the beginning I would have done. Green field projects can, we can't. But instead we have to migrate. Its a drag and I would love to be on 3 but we have to take baby steps as we must also maintain function.

Hope that helps and do take a look as our outstanding issues and the like as we have had, as stated, many first time coders improve little but important elements of our existing code without need to re-write all that already existed. But our OS change was quite the challenge as it goes. But at least going forward, post the Python changes, we can again adopt our dual upstream OS arrangement of Leap and Tumbleweed. Of far more interest and value to most than our Web-UI framework (as long as it works mostly that is ;).

Also note that the two lead developers both use Pycharm. It's fantastic at highlighting, and tracking down definitions etc. And has some nifty db capabilities in the Pro version. Even better once we get to Python 3 of course ;). My frustration with "just re-write in" suggestions is likely evident here. And you may well have come across this same suggestion in your own projects. And if not you likely will in time. I've had it a few times here over the years already. :) And my predecessor, the project found @schakrava had likewise for a few years!

What is needed is effort in assisting with our project goal of updating our dependencies, now we are on our new OS. This would be much appreciated. And if you can define one via an issue, or adopt an existing one, and address it in testing then dandy. This would be much appreciated. And likely very well received. We are actually super friendly, as is evident form our many first time contributors. But we must move in baby steps as we are trying hard to maintain function. And this is complicated by such major technology changes so we have to go bit by bit. Yes we have a lot of old dependencies. But at least our users have a modern kernel and OS now to manage their data!

And yes, we need a road-map (recently discussed but not finalised in side channel wouldn't you know). Here you go:

1. Django update (mostly done and merged, just need 2.)
2. Prove function (the testing issue above).
3. Re-start engagement with users for our ongoing field testing via a new testing channel rpm release phase)
4. Python 2 -> 3 (enabled by 1., 2., 3.)
5. New build system (enabled by 4.)
6. Look to our long neglected Apparmor incompatibility (this can be done at any time actually).
7. Ideally a newer Django again, LTS should be refreshed by then (enabled by 4.)
8. Establish openQA end to end testing from installer though to all base functions (POC in the works).
9. Re-introduce Tumbleweed as a target OS as it was invaluable as a heads-up for OS technology changes (enabled by 4.)
10. Begin our far more exciting plans to support GlusterFS, multi-path, iSCSI, btrfs raid1c3 raid1c4 raid56J etc.

All doable and hopefully while actually providing wanted feature expansion. I see buildbot at our example of possibilities here. It was a massive project that dealt with it's technical dept in time (also Python 2 to 3 mostly). We must do the same, bit by bit and in time. Our next testing channel phase should address the recent Django update fall-out and return us to proven capability via our test set. And our consequent testing channel rpm releases should ensure our continued actual field-use function, not the theoretical abstraction that is smaller unit tests. We also have, in the works, an initial proof of concept for 7. But our concerned developer on that front is, as is so often the case, involved in other matters of consequence also :). I believe he is instrumental in saving lives somewhere or something (Hello gain @FroggyFlox).

I myself am temporarily pre-occupied with some background business concerns essential to the greater project goals of sustainability in open source development. But once sorted on that front my own contributions should return with gusto. Assuming I'm not demotivated by repeated calls to rewrite everything from scratch :). Rust has been one by the way. Go another. Both super interesting but still! Also note the current difficult in getting Rust and Go developers.

So I am wondering if the core team has discussed any of this and what were their thoughts / decisions. Is there a discussion or a place to start this discussion?

As per our docs: our friendly forum is a good place to start anything friendly. Then we try to spin of doable discreet tasks that are as exactly defined as possible, in order to be easily actionable.

https://forum.rockstor.com/

Hope that helps.

[FroggyFlox]

Most of the detailed modernization has now been completed, and more is currently under way. In an effort to tidy up our issue tracker, I'm thus closing this issue so that we can better identify focus points moving forward.