top: Reorganise micropython-lib

[jimmo]

One long-standing point of confusion around this repo is what it's actually for. The answer is that it's for making the Unix port be more like CPython, but that is not at all clear without really diving into the repo.

This PR aims to re-think this module instead as a common repository of useful modules, with the unix/ffi bits out of the spotlight. I'd eventually like to see this become a centralised place for MicroPython users (on all ports) to find drivers and useful libraries, and possibly as a staging ground for candidate libraries for freezing into some firmware build.

So this PR:

• Removes any empty placeholder libraries (they're not helpful, and it's not clear that they're empty without some work)
• Removes any libraries that are no longer useful (e.g. asyncio_slow)
• Moves all libraries that require the Unix port into the unix-ffi directory.
• Separates MicroPython-specific and CPython-compatibility libraries into the micropython and cpython-stdlib directories.
• Adds new READMEs.

I'd like to volunteer as a sort of curator/maintainer role, especially with the goal of fostering library development for libraries that automate or simplify common tasks on MicroPython. Here's a random example off the top of my head: if I'd like to build a wifi-enabled widget, I'd love to come here and find a library for each of the tasks like captive portal, ap-setup-mode, save-credentials, etc. Similarly some high-level wrappers around common tasks with ubluetooth. Or pin debouncing, or drivers, etc.

The big outstanding TODO is what to do about upip / pypi / etc. Before I tackle that I'd like to see what people think of this approach.

The other option is to just deprecate this repo (i.e. re-revert the "this repo is unmaintained" commit) -- let libraries be developed and distributed independently and perhaps consider only bringing really core useful libraries into the main micropython repo. I'd prefer to see a centralised repo, I think that's good for the community and good for discoverability.

[jimmo]

I should add one other note -- there are a few libraries in the python-stdlib directory that I'm not particularly interested in maintaining and would probably be keen move (archive) into the unix-ffi directory. These mostly fall into the category of "very unlikely to be used on a microcontroller". But first things first.

[dlech]

It would be nice to see this repo actively developed again.

I like the idea of having different categories/folders for code with different goals.

[andrewleech]

I like the idea of getting rid of the stubs, I too don't feel they're particularly useful at all

[andrewleech]

As far as upip is concerned, I'd recommend starting our own themed warehouse (https://github.com/pypa/warehouse) server that upip looks to preferentially, then people can submit packages to that directly with the same tools as pypi.

This repo could then be set up with a ci build to automatically push any changes in the included packages directly to uPyPi

[mattytrentini]

As far as upip is concerned, I'd recommend starting our own themed warehouse (https://github.com/pypa/warehouse) server that upip looks to preferentially, then people can submit packages to that directly with the same tools as pypi.

This would be fantastic. I know Damien already has a custom 'PyPi' server in place but we should use warehouse so the community can help maintain the collection.

[mattytrentini]

That all sounds great @jimmo - and having you as a curator makes a lot of sense.

A few extra notes:

• Consider putting unix-ffi in a separate repository. I think it's only of interest to a very small user base and confuses the majority of people using MicroPython on embedded devices.
• It would be fantastic to have some way of clearly seeing which libraries implement standard library features. I think we could nudge the community into helping fill the gaps in standard library support if it were clearer what to work on. Bonus points if we have a coloured table showing which methods for each standard library have been implemented for MicroPython.
• Possibly consider whether submodules for each library makes sense. CircuitPython uses this approach and it scales well for the community. So micropython-lib would be a collection of submodules.
• Documentation will be key; it should be very clear as to how someone can contribute or create a library.

But yes, reinvigorating micropython-lib is a fantastic idea!

[tve]

💯
Also 👍 on "Consider putting unix-ffi in a separate repository"

[UnexpectedMaker]

I think this sounds great and a big thanks to @jimmo for volunteering to curate!

I'm unclear about who would be responsible for maintaining the community submitted libraries? The member that submitted it? Or does it get "taken over" by the curator and MP folks?

Also, this is going to sound pedantic, but as someone that has had their code "changed to be more like how someone else wants it to be" a few times now, that also has more libraries that could really be useful in this repo (WifiManager anyone?) the "you should not be using properties", "functions need to be formatted like this", "don't put spaces here or there", "text should be formatted like this", etc really puts me off wanting to contribute.

I know this really doesn't have anything to do with this specific PR, but it kind of does as I think Jimmo jumped into this PR after we chatted yesterday while he was helping me with my WifiManager / asyncio issues. My WifiManager was suggested to be an ideal candidate for inclusion.

One of the biggest issues with using MP right now is the lack of libraries, and that void will only get filled by community folks contributing, and if contributors are left to feel like their code "isn't good enough" or "doesn't match the repo rules" then we are going find folks might not want to contribute.

Happy to be smacked over the head for this... but it is truly how I feel :)

[UnexpectedMaker]

I should add that I am super excited that some (any) momentum might happen with regard to maintaining a good/large curated set of "official" libraries.

[stinos]

This implements pretty much everything I mentioned in Slack so I'm very much +1 on the core principles :) The actual implementation is going to take some discussion probably.

I'd like to volunteer as a sort of curator/maintainer role

You mean you want to be maintainer in the Github sense (accept PR/push into master) of this repository together with Damien?

Consider putting unix-ffi in a separate repository.

@mattytrentini I'd actually consider splitting all roles into separate repositories (stdlib/micropython/unix-ffi). That also addresses concerns like 'It would be fantastic to have some way of clearly seeing which libraries implement standard library features' a bit better than just a sub directory. And it's also easier to tell to people: 'just go there for the std lib we have', without having to worry they get lost (as is the case now, in non-existing/non-conformant/not-working-on-your-port modules).

Especially if the plan is to open up the micropython part for all sorts of user contributions, I think the stdlib should be kept separate.

consider whether submodules for each library makes sense.

There's a lot of them though. What would be the main benefits which make the extra maintainance worth it? Also that basically means each issue/PR has to go in it's own repository? For the micropython part of the library that could actually make sense though.

"text should be formatted like this", etc really puts me off wanting to contribute.

While I get that, you'll have to live with it. Consistency is key for projects with a lof of contributers. I've worked on projects where nearly every file looks different, not to speak of the formatting diffferences within each file. It's exhausting. It makes reading harder (that's actually a fact, these higher cognitive loads have been studied and published). It's annoying because if you want to add code you have to decide whether to use your standard style, the style of the surrounding lines or just pick something at random. And so on.

And as far as formatting goes there's now tools/codeformat.py so it's easy to get right.

[stinos]

@dpgeorge any thoughts on this? It would be good to move to move forward with micropython-lib as it's a major tool in getting more CPython compatibility.

[mcauser]

Consistency is key for projects with a lot of contributors.

Providing the tools / scripts / docs to keep the code consistent would help a great deal here. I wouldn't mind my code being slightly rewritten to match conventions, as long as said conventions aren't constantly changing and adding heaps of git diff noise.

[stinos]

I think we're basically on track for that: codeformat.py takes care of most of that both for C and Python code (it just needs to be used for this repository as well), and before it existed the code conventions were also followed rather well and not a lot has changed over the course of the years. Ok thare are some edge cases but unlike many other projects out there I feel like microPython does a very good job when it comes to consistency.

[CleberPeter]

I am new to micropython development. And as a new user, I can agree with the above discussion, the confusion is great.

I checked this repository for the existence of some PR open for some time without completion, however, the esp32 manifest still includes modules from this repository:

https://github.com/micropython/micropython/blob/master/ports/esp32/boards/manifest_release.py

Anyway, is the contribution to this repository encouraged or is there another one that you can indicate?

[dpgeorge]

Thanks @jimmo for this proposal. I am fully supportive of it. (And sorry for taking forever to respond here.)

I agree it makes sense to distinguish between conceptually different sets of libraries such as:

• Ones aiming to extend CPython stand library modules that are in the core/main repo, to provide missing functionality; eg os.
• Ones aiming to implement exactly a CPython standard library module that is not implemented in the core; eg logging.
• MicroPython specific modules, eg a display driver (could possibly include native code).
• Modules intended only for use with ffilib; eg sqlite3 (I guess in general this category covers modules that are very port-specific, not just unix ffilib ones).
• Third-party modules, which could possibly be a git submodule.

But even if there is such a separation into categories like this, it's still important that each module have a globally unique name so any two of them can coexist in the same system.

I'd prefer to keep everything in the same repository, rather than split them up into separate repositories. I just think it's easier to manage that way.

[stinos]

I agree it makes sense to distinguish between conceptually different sets of libraries

Just to be clear: you mean a distinction by means different directories, or something else?

[dpgeorge]

I agree it makes sense to distinguish between conceptually different sets of libraries

Just to be clear: you mean a distinction by means different directories, or something else?

Yes by different top-level directories, rather than different repositories. I think different repositories makes it harder to manage issues, and more work for users to find the correct repository / search.

[dpgeorge]

Merged! Thanks @jimmo !

[graham768]

I'm late to this party and new to MicroPython as a whole, but I'm having trouble understanding where sdist_upip now lives. It's referenced a few dozen times throughout this repo in the current master branch and was removed in this PR (and doesn't seem to have been included in the micropython/micropython repo either).

The link in the official MicroPython documentation is dead and other projects I've seen have simply grabbed their own copy of sdist_upip.py as a top level file from wherever they can find it -- is this the expectation now?

[dpgeorge]

but I'm having trouble understanding where sdist_upip now lives

It's been removed and replaced by mip. See http://docs.micropython.org/en/latest/reference/packages.html