[Idea] Online "make-your-own-OS" build tool

[adriweb]

Considering that more and more suggestions are about (or getting categorized as) "build-time options" (because why not, it could save space for people not needing them), I've been thinking about having an online service (builder.numworks.com ?) that users could use to customize & download their own build (in addition to being able to download official/popular ones).

The idea itself is rather simple and would work like existing services that do so (for instance in the world of web-dev: jQuery UI, FabricJS, Initializr...):

• The user is presented with a list of togglable features to be built, some enabled by default (for the default/official build).
• Once the user is done choosing a set of options, he submits the form and the server builds it and serves him the firmware file.
• Custom/popular sets of options would be directly available making downloads instantly available.
• We could even imagine emscripten builds to be done on the fly so that the user can actually try an app/feature online before embedding it in his "distro"

Where things get a bit more complex (but not so much, really) is that there'd be a server responsible to receive build orders, make them (shouldn't really take more than a few dozens of seconds max. on a decent enough computer) and serve those builds somewhere (adding to the list of available builds in the set of options, so the next person selecting those same options will have an instant download).
Depending on the number of options available (might be just a few initially), it would probably be worth it to just generate all the possible combinations, so all downloads are ready.

Until (if it's ever going to be the case) there's dynamic code loading, this could very well be some kind of [not-so-]poor man's "App Store": on the developers' side, features/apps finding their way into online builds, and on the users' side, easy usage of a customized OS as they like it without having to download a toolchain/sdk/etc. to make it themselves :)

This also allows to see which optional features are the most popular.

[Zezombye]

Would also be a good compromise for this : #42

There is no need for a file system or run-time addins, and since you don't need to compile the OS yourself you don't need to be tech-savvy.

[adriweb]

Yep, definitely - edited.

[Ecco]

Actually, this is most likely where workshop.numworks.com is heading :)

[Ecco]

On that matter, here's a question for the community: besides Epsilon itself, that would also be a great opportunity to also distribute apps that were deemed unfit for inclusion in Epsilon itself, just like @Zezombye suggested.

This raises the question of API/ABI stability, but let's put that matter set aside for a moment.

My question is: should we encourage developers to distribute their apps as source code or as binaries?

Source code

• useful, everyone can see how the apps are made
• removes the problem of ABI altogether
• but can be a pain to build, we may not want to enforce our build system for each app
• maybe some developpers won't be comfortable sharing their source code

Binaries

• super easy to distribute
• would need a lot of different builds (device, simulator, web, etc…)

[adriweb]

Excellent.

Regarding apps distribution, I'm pretty sure developers making NumWorks apps would be open-source enthusiasts, so I wouldn't worry too much!
Binaries are indeed much simpler for the end-users but do pose an ABI compatibility issue - not that it's much of a problem for open-source apps that would get built "on-the-fly" by the builder though :)

[Zezombye]

Binaries would be required for non tech-savvy users, however source code is recommended because why be closed source?

[adriweb]

Well, simple compromise: open-source apps could be integrated into the online builder thanks to the source being available (for rebuild on the fly* and binary download as soon as it's available), while non-open-source ones get their binaries distributed by the developer on his own platform.

* I'm not sure it'd be such a pain to have some minimal template to follow?

[Zezombye]

I don't think it would be a good idea to automatically build the files, because some developers might use a custom makefile or SDK. Distributing the binary and uploading it on the workshop would be enough, and if an update breaks the app, then it's up to the developer to rebuild it - or even the community, thanks to open source.

[adriweb]

Distributing the binary and uploading it on the workshop would be enough

That makes the whole "choose your own features/apps" not working anymore though - remember we can only have compile-time things. Unless somehow some reliable enough format of some .a library gets uploaded by the developer and used during the build, but that doesn't really seem safe and brings up the ABI compatibility issues.

[Zezombye]

Indeed, if it is possible to build the app automatically using a makefile then it would be enough. Also, there would need to be a sdk with native functions (but that's another issue).

As for the file system: wouldn't there need to be one for saves? How else would games save progress?

[boricj]

I would go for encouraging source code. You can't integrate binary apps without a linker anyways.

[rhaamo]

File system is another issue, as is the "external apps in binary form".
The filesystem would be accessible using the API offered by the "Apps SDK" anyway, like Lowlevel/drawing/calculating etc. would be.

I think the first step is to consolidate what's existing, like at least having modularity for compile-time, a proper documentation, and then it should be possible to think about what to do.
We don't even know if NumWorks plans one day to add the uSD slot or the flash one, so...

[adriweb]

Yeah, some open-source architecture is basically the only good way to go so far...
And I don't think this issue is the place to talk about a FS, there should probably be some brainstorming elsewhere ;)

We don't even know if NumWorks plans one day to add the uSD slot or the flash one, so...

I mean... both are in the schematics already, and there is source code for uSD support already. It's "just" that it's not sold with those devices soldered in :P
(Note that the French government forbids the use of SD cards - or any other object that you can "stick in" the calc)

[rhaamo]

I mean... both are in the schematics already, and there is source code for uSD support already. It's "just" that it's not sold with those devices soldered in :P

for the moment, and it will only target a small part of people actually (the one who can do the upgrade, and void the warranty :p)

(Note that the French government forbids the use of SD cards - or any other object that you can "stick in" the calc)

If you need to open (4 screw) the calc to add only one time the uSD, I think it can be ok, technically they forbid you to literally open the calc body in an exam, and an internal uSD accessible using a screwdriver would be hard to be considered as an "external peripheral" :)

[Zezombye]

File system is relevant here, because on #42 the argument against runtime apps was that numworks would need to develop a file system, which could take time. However, since we should implement a file system (for saving and other things such as custom levels) then why not add runtime apps anyway?

Maybe I'm wrong, but I think this is simple: all they have to do is to execute the code in the app, then maintain syscalls for IO.

[boricj]

Here's a (non-exhaustive) list of things needed to support third-party native apps at run-time:

• A way to store things at run-time ;
• A way to transfer apps to the calculator at run-time ;
• A way for apps to interact with the firmware ;
• A third-party app (obviously).

First point needs a file system in Flash because making one in RAM with dynamic memory allocation is not going to cut it. Second point needs USB data transfer, which is not a trivial thing to implement, otherwise you could add apps only by flashing a new firmware. Third point needs syscalls or dynamic linking.

The first two points would be very useful beyond third-party native apps, yet they are completely unimplemented for now. I think it is fair to say that it is simply too early to worry about that, the firmware couldn't even transfer and store those hypothetic native apps for now anyways.

[debrouxl]

Indeed, for the reasons explained by JB, third-party native apps selectable at run-time can only be a long-term goal... Compile-time selection, which is the goal of his #73 , is a much more achievable goal - and then there can be some focus on easing that task.

The prerequisites for run-time apps to be possible should obviously be kept in mind for prioritizing, designing and coding the next steps on the firmware, but they look out of scope for this issue, don't they ?

I think we should create a number of other public issues for tracking at least the following items:

• defining some mechanism for two-way interaction between FlashApps and the core OS (which could include a bit of lifecycle information, e.g. "starting" and "stopping" the FlashApp);
• maybe defining some mechanism for one-way interaction (based on events ?) between FlashApps;
• adding filesystem support (for both RAM and Flash);
• representing internal variable data inside the filesystem;
• providing USB communication for file-type data transfer purposes;
• providing USB communication for other data and control purposes (I mean, operations which don't easily lend themselves to pure file-based communications, such as remote control, which have been a standard feature of TI calculators for a long time);
• making computer-side software for communicating with NumWorks calculators and dealing with variable data;
• making a computer-side accurate emulator for the NumWorks calculators.

However, I think there should be some more brainstorming and collaborative description editing before posting issues with a proper description on Github :)
Obviously, we'll collectively reuse our knowledge of existing calculators, protocols and corresponding computer-based software in the design and coding phases, so that we don't have to reinvent the designs of wheels from scratch, "only" adapt and hopefully improve them.

[Zezombye]

It is a long term goal, however it is useless to build a "build your own OS" tool, if you're eventually going to add compile-time apps.

[adriweb]

No, because not only would apps be one (and not the only) thing that the user could toggle *, but also the timeline isn’t the same at all : compile time selection & tool could take days/weeks, while run-time things would be more like weeks/months to do correctly.

* Examples: features, settings, themes, translations... those aren't apps but are worth being set at compile-time.

[boricj]

Indeed. Also, there's a lot of functionality yet to be implemented as a calculator according to TI-Planet's QCC when compared to other models. For now, I would very much prefer the team to focus on making NumWorks more attractive to the bulk of potential buyers (students who need to solve problems with the help of a calculator) so that they may survive in the marketplace rather than use up dev time on use cases that won't yield nowhere near as much return on investment.

Compile-time apps won't take much time or focus off of this and it will fulfill the majority of the calc dev community needs. Combine it with an online firmware building tool and it will allow nontechnical students to use our homebrews with ease. Run-time native apps will require much more work for not much more results.

The NumWorks team can't afford to work on run-time apps right now or in the near future and there's no way the community will find a non-intrusive, maintenance-free way to pull it off and mainline it. At least wait until they establish a sustainable position in the marketplace that guarantee their survival before sending them on a wild goose chase.

[Ecco]

Thank you @boricj

[0b101]

[boricj]

@0b101 This layout will not scale. The whole point of a make-your-own-OS tool is the ability to include third-party stuff that is not inside the epsilon source tree (like https://github.com/boricj/numworks-tic-tac-toe), otherwise unless NumWorks runs out of flash space there's little reason for normal users to downsize their flash footprint by removing features. So for apps, this means a list of selected apps as well as a way to browse/search apps (and the same for languages).

@Ecco While I'm here, there's a technical issue with third-party languages: the localization files are mangled with the epsilon source code. We can't stash them in an independent subrepository like apps. This is a problem especially because currently there are language translations PRs (#190, #438) that are bit-rotting and it's now fairly obvious NumWorks will not mainline community-provided languages. Maybe we should move i18n files to a dedicated directory per language (so that we can maintain out-of-tree translations in repositories), but we also need to keep the ability to bundle translations with apps for third-party apps...

[0b101]

@boricj This layout was designed with scaling in mind. There will be an "upload" button either in the step or on a separate "add" page. The design can easily be hooked into a PHP script with or without a database of know sources. Submission of new apps/changes can also be handles with a PHP script and a database.

Maybe we should move i18n files to a dedicated directory per language (so that we can maintain out-of-tree translations in repositories), but we also need to keep the ability to bundle translations with apps for third-party apps

Third party apps could supply the translations for the default languages and set a certain language as default if the system's language is not one of the 5 defaults.

[boricj]

I'm still not convinced a sea of buttons is the best UI for this, it just doesn't feel neat at all. What I mean is, NumWorks prides itself with a clean, modern and easy-to-use graphical interface. Might as well spend some more time researching a good, neat UI/UX combo mockup.

Also, the user will most likely want the ability to order the apps on the home screen in a certain way so the concept of an ordered list is important. Personally I'd start with a left pane with a vertical list of apps to include (with a trash icon) and a main right pane where you could search/browse apps and maybe check out the details of an app (its README.md?) by selecting it, something along the lines of Apper or the Ubuntu Software Center. It's certainly not the only way to approach this though.

By the way, a button for uploading apps will work as long as you're uploading an archive of source code (like https://github.com/boricj/numworks-tic-tac-toe/archive/master.zip). Not so much for translations for the time being, which we should probably fix at some point.

[LeJoW]

I really don’t know how works the programmer side of a linux distribution repo (typically used by apt-get) but I think it's a good way to share apps, isn’t it ?

[paternal]

I really don’t know how works the programmer side of a linux distribution repo (typically used by apt-get) but I think it's a good way to share apps, isn’t it ?

Or repositories of programming languages (http://pypi.org, http://cpan.org, http://ctan.org, etc.) : anybody can upload (free) libraries there, and there is a (more or less) user-friendly way to install (and update) them on the calculator.
This is quite a lot of work for the Numworks team, though… 😛

[theMackabu]

something like https://zardam.github.io/webnofrendo/ right?

[boricj]

@sajjaadf This merely concatenate an already-compiled NumWorks firmware with a NES ROM, it does not allow nowhere near the level of customization discussed here (we can't pre-compile all possible permutations either because there's way too many of them). At the very least we need to run a linker to tie together pre-compiled apps and chunks of firmware.

[theMackabu]

I meant the WebDFU idea

[adriweb]

I meant the WebDFU idea

https://ti-planet.github.io/webdfu_numworks/ is something I made quickly based on this that allows users to more easily dump and flash a firmware.
But you'll need to make the firmware yourself anyway in the first place (or use another tool that does it), that doesn't change.