tools/micropip.py Add CPython tool to install libraries on a PC.

[peterhinch]

The official way to install library modules via upip is great for networked hardware. For users of other hardware this means cloning the MicroPython source, acquiring the compiler and building the Unix port. It also requires a Linux (real or virtual) machine. These are significant hurdles for users familiar with Python but lacking experience of Git, Linux or C compilation.

This tool runs under CPython 3.2+ and is cross-platform. It is based on upip with the same invocation but uses standard CPython libraries and OS-agnostic path manipulations. Testing was on Linux by myself and on OSX and Windows by users.

If this PR is accepted I'll submit a PR to update the library README.

[pmp-p]

well done ! maybe you could have a look to https://github.com/pmp-p/micropython-ffigen for some kind of "wheels". if micropip could setup/build/install C extensions that would be great, or even prebuilt as some platforms need cross compiliing which is awfully complicated sometimes.

[dpgeorge]

Thanks @peterhinch for submitting this. I just tried it out and it works well! I installed from upip directly to a mounted pyboard.

It is based on upip

Yes I see that it's quite similar, and so your code is essentially a version of upip.py that runs under CPython. I can't help to think that a better solution (eg would require less maintenance in the future) is to make it so that upip.py can run under both uPy and CPython.

Doing a diff of micropip.py with upip.py show some significant differences. Is it really too difficult to make one script for both interpreters?

[peterhinch]

The upip code looks to be optimised to run on resource-constrained platforms, notably the ESP8266 via WebREPL. I was therefore reluctant to make any changes to it. These would inevitably increase its size and complexity.

[pfalcon]

As the author of the upip, of which this "micropip" is a slight modification, and otherwise a developer of module ecosystem for MicroPython, I'm strictly -1 on this for the following reasons:

1. This submission impersonates the authorship of the tool.
2. The functionality is already provided by MicroPython, which is full-stack system otherwise. This submission promotes the notion of something like "MicroPython users which don't use MicroPython" which is oxymoron.
3. This is a retrograde action overall, we started like that, using CPython's "pip" tool to install MicroPython packages: 802d505. A lot of effort went into implementing native package manager for MicroPython. Then finally the CPython wrapper was removed in b78144c.

[peterhinch]

@pfalcon I have no intention of "impersonating authorship". If the form of words I included in the source doesn't meet your requirements please submit one which does and I will happily use it.

At the time when I ported the code the source had no attribution or licence details: I emphatically did not remove them. I am no lawyer but (to refer to your comment in the forum) I fail to see how I can be accused of "violating the terms" of a licence not specified in the code.

When porting code I always include attribution if it exists.

As stated above I will be glad to remedy this as you wish: a polite PM rather than public accusations would have sufficed.

[stinos]

This implementation does have the merit that it just works on whatever OS, now. Whereas a full 'native micropython' solution for upip (as you mentioned in another thread) would either need at least one of the windows ports to have a socket/ssl/... implementation and changes to path handling.

But I also think this could use some clean up. Maybe in the form of a common module which shares the common code with upip. And/or maybe if using CPython anyway it seems wiser to reuse it's makedirs (and possibly other functions') implementation. Likewise when using CPython there's a debugger so the commented out print statements seem superfluous.

[pfalcon]

I have no intention of "impersonating authorship".

Of course, all that you do (and humans in general) is based on good intentions.

At the time when I ported the code the source had no attribution or licence details: I emphatically did not remove them. I am no lawyer but (to refer to your comment in the forum) I fail to see how I can be accused of "violating the terms" of a licence not specified in the code.

So indeed, this was brought up on forum: https://forum.micropython.org/viewtopic.php?f=2&t=5540&start=20 , with detailed discussion. The claim that the code had no attribution is not true: the code is part of the micropython-lib project with explicit LICENSE notice/instructions: https://github.com/micropython/micropython-lib/blob/master/LICENSE and that particular code is accompanied by licensing/attribution file https://github.com/micropython/micropython-lib/blob/master/upip/setup.py . The code otherwise didn't have builtin header exactly because of caring for people with pyboards and other things where space is scarce. The idea was that people would pay due diligence and respect to all that.

And you don't have to be a lawyer to know and understand that inaction can be as a wrongful act as an action, based on the consequences (though "I didn't do anything" indeed would be a common excuse).

As stated above I will be glad to remedy this as you wish: a polite PM rather than public accusations would have sufficed.

I have a suggestion for you to: please pay due respect and diligence to work done by others.

[peterhinch]

Sometimes I think I should take up golf instead 👎

[pfalcon]

This implementation does have the merit

Of course, everything has its merit. And its place, too.

that it just works on whatever OS, now. Whereas a full 'native micropython' solution for upip (as you mentioned in another thread) would either need at least one of the windows ports to have a socket/ssl/... implementation and changes to path handling.

So, when there was a need to develop a native package manager for MicroPython, I got to work and implemented it. What response would you, a Windows maintainer, expect to a complaint that there's no ssl, etc. implementation exist in Windows port?

MicroPython would never exist if there was such "that can be done with CPython instead" "revelations". Likewise, it unlikely would get far beyond blinking-led state with ideas like that.

So yeah, I find it highly ironical (and much frustrating, if that reflects back to the main repository), that someone takes upip code (which also happens to be written by me), carefully forgets to mention that its based on upip/written by me, and uses it for exact the opposite purpose than it was originally written (to make MicroPython self-contained, not dependent on CPython).

[pfalcon]

Sometimes I think I should take up golf instead

Oh, you could do a lot of things driven by good intentions! For example, could crash on AA sessions with a few bottles in your hands. Who cares that someone tries to get them to higher spirits? Why subject those poor people to all that effort? All their problems can be more easily solved by just giving them free booze.

[peterhinch]

I have no investment in the code, but I stand by the use-case 100%.

The port took a few hours work and I claim no originality whatsoever. If you, or anyone else, cares to write a better version, I will remove this one and point forum users to its replacement. As stated above I will replace my legalise/attribution with text of your choice: please advise.

I'm not sure what more I can offer.

But if this vituperative rather than cooperative approach is the future of MicroPython, the golf course beckons.

[pfalcon]

I have no investment in the code, but I stand by the use-case 100%.

Likewise, I stand by the usecase that MicroPython is a full-stack ecosystem, and users should be encouraged to view, use, and elaborate it further as such. Instead of diluting that by "easy" workarounds. And I stand by that with all the amount of contributions I made to MicroPython.

please advise.

Once you comply with the license terms, you're welcome to maintain and promote "micropip" as your personal project. Well, if you want to make me fully satisfied, add a notice that I, as the author of the original code on which it is based, discourage its use and encourage use of the original upip instead ;-).

[stinos]

Sometimes I think I should take up golf instead

Code golf? :P

What response would you, a Windows maintainer, expect to a complaint that there's no ssl, etc. implementation exist in Windows port?

Well, I looked into implementing sockets already at one point. Don't recall why but it wasn't too straightforward. And that's just sockets, no ssl. Add to that the limited amount of interest (i.e. none, back then) and there's the reason for the lack of it currently. Now if the consensus here really is that upip must not be changed to also work with CPython (which I think is still better than having 2 seperate implmentations and probably still the most versatile solution for all ports and platforms combined) I might reconsider.

[pfalcon]

If you, or anyone else, cares to write a better version

Note that with the latest developments in micropython-lib https://github.com/pfalcon/micropython-lib#cpython-backports , it would be possible to run upip directly on CPython - once suitable module are ported of course.

Oh, and let me mention, that once matters like #4195, #1627 are resolved, there would be a room indeed for a "meta-tool", which does much more than just upip.

So, no easy "free booze" solutions, all require a lot of effort - aligned effort, based on the directions the projects goes for years. Instead of taking "easy" shortcuts, effectively going against the direction of someone's work.

[peterhinch]

I have included this text in my personal version. If this is satisfactory to you, and there is any likelihood of this PR being accepted, I will amend the PR accordingly.

Otherwise please post the exact text you wish to see and I'll implement that.

I understand your reasons for disliking it; in many ways this is a re-run of your distaste for my old scheduler. In the case of micropip, at cost of a few hours work it fixed a practical problem encountered by a number of users. I'm an engineer by training and see merit in quick solutions to practical problems. A better solution delayed by a few years is, er, better, but doesn't help beginners who are struggling now.

[pfalcon]

If this is satisfactory to you,
Otherwise please post the exact text you wish

To state the obvious, it's a license, the formal matter. You don't need to ask me how to do it, just do it like others do it (which includes doing some research and spending some effort each time you take somebody else's code for redistribution/modification).

To answer the specific question addressed to me: yes, that looks good, given that you have LICENSE file in your repository, which provides the full text of the license.

I understand your reasons for disliking it; in many ways this is a re-run of your distaste for my old scheduler.

I'm not sure if you do. So, let me try to formulate it: for the amount of time you use the project, and the amount you work to do with it, I regularly find that the outcomes you share are ... well, not thorough enough. Obviously, that's highly subjective and who am I to judge. But given that you're also very keen to "spread the word", for cases that happens to be true, that indeed may lead to skewed effects, as many listeners are even less critical of the information you provide than yourself.

I'm an engineer by training and see merit in quick solutions to practical problems.

Here's one case. Many engineers know that "quick solutions" are oftentimes nothing but workarounds, which can easily have detrimental effects. But you announce that as a "solution".

A better solution delayed by a few years is, er, better, but doesn't help beginners who are struggling now.

Then start doing it now - right, and it won't take a few years (we have bad counter-examples, but those are exactly that - antipatterns). And there's always an options of having had started it before so it would have been done by now.

With all that, it's just your opinion against mine. I don't have any specific influence on decision making here now, except for discussions like this, where I can elaborate how and why something was done, and how something else doesn't align with that.

[pfalcon]

$ pip3 install --user micropython-cpython-upip
Collecting micropython-cpython-upip
  Downloading https://files.pythonhosted.org/packages/23/86/7a98c5b0840d65ffbe6e71a0a9bd2ab93f5c938542b84a3c4824d2fc1553/micropython-cpython-upip-1.2.6.tar.gz
Collecting micropython-cpython-uerrno (from micropython-cpython-upip)
  Using cached https://files.pythonhosted.org/packages/55/65/06f724152cc383704b44a92a2d4dc8f62159f64c6bbb84e21f83b62aeb28/micropython-cpython-uerrno-0.1.tar.gz
Collecting micropython-cpython-ujson (from micropython-cpython-upip)
...
Successfully installed micropython-cpython-micropython-0.2 micropython-cpython-uerrno-0.1 micropython-cpython-uio-0.2 micropython-cpython-ujson-0.1 micropython-cpython-uos-0.1 micropython-cpython-upip-1.2.6 micropython-cpython-usocket-0.1 micropython-cpython-ussl-0.1 micropython-cpython-uzlib-0.1

$ python3 -m upip install -p install_dir notes-pico
Installing to: install_dir/
Warning: pypi.org SSL certificate is not validated
Installing notes-pico 0.8.9 from https://files.pythonhosted.org/packages/93/d3/f15081396c45b77709c9346215a45836130ae0367e801d17bb69a19751a3/notes-pico-0.8.9.tar.gz
Installing picoweb 1.5.1 from https://files.pythonhosted.org/packages/56/5c/4540e02d8be5b4dc3cc705a3047fb54c1d79e98f49b115290d7a2e736f5d/picoweb-1.5.1.tar.gz
Installing utemplate 1.2 from https://files.pythonhosted.org/packages/ce/c3/6ef0cc5c256c2192d8f6ff0175a6b6c3dff727fdb6ec1eb432fc0a2a97e9/utemplate-1.2.tar.gz
Installing micropython-logging 0.4.1 from https://files.pythonhosted.org/packages/46/93/7ce7c6632c59a47908b068d047d7b76c4505c6bf2b2dfa53b540fb1881bc/micropython-logging-0.4.1.tar.gz
Installing micropython-pkg_resources 0.2.1 from https://files.pythonhosted.org/packages/58/87/5acc262e0f642186891ef8d944bd727989000ce8d9263514f73b8feb7fdc/micropython-pkg_resources-0.2.1.tar.gz
Installing micropython-btreedb 0.3 from https://files.pythonhosted.org/packages/c4/03/b60fdda40235da6322ce11948a2576da7a2d8c4fe56d31f4440388e49a59/micropython-btreedb-0.3.tar.gz
Installing micropython-uasyncio 2.1.1 from https://files.pythonhosted.org/packages/0c/33/cbd1ee245621dab9952ebd1d3be840e6acd66e847b33ea88a7f015d2b952/micropython-uasyncio-2.1.1.tar.gz
Installing micropython-os 0.7.1 from https://files.pythonhosted.org/packages/a7/dc/9956b7ea7e4cc46e3d2a326225a2ea48fe66bcbea03f7ac0c84519445c4d/micropython-os-0.7.1.tar.gz
Installing micropython-uasyncio.core 2.0.1 from https://files.pythonhosted.org/packages/39/9f/619bbe7454419b72073e61b3d9abbc5dfeb54cc4062c2a2d21ad4984db85/micropython-uasyncio.core-2.0.1.tar.gz
Installing micropython-ffilib 0.1.3 from https://files.pythonhosted.org/packages/81/8a/5c3e6188384fbb57800a188d24582dc1a16b4fc8b538e0ba58aa443e3573/micropython-ffilib-0.1.3.tar.gz
Installing micropython-errno 0.1.5.1 from https://files.pythonhosted.org/packages/c5/bc/ac0b6e27a37f985ac5237ad1a6633a89a646cb23152c27fecca368abe811/micropython-errno-0.1.5.1.tar.gz
Installing micropython-stat 0.5.1 from https://files.pythonhosted.org/packages/cf/cf/fde35e109408fb94a5c00725e53608dada21a6b3eaa04dfd4189a8def2c1/micropython-stat-0.5.1.tar.gz

$ MICROPYPATH=install_dir micropython -m notes_pico.__main__
* Running on http://127.0.0.1:8081/

[peterhinch]

Nice solution and I'll happily can my port and this PR when I can make this work.

I'm having problems. The first stage was successful but the second produced this outcome under CPython 3.4/Debian 8:

root@debian8:/home/adminpete# python3 -m upip install -p /home/adminpete notes-pico
Installing to: /home/adminpete/
Warning: pypi.org SSL certificate is not validated
Error installing 'notes-pico': TypeError("the JSON object must be str, not 'bytes'",), packages may be partially installed
Traceback (most recent call last):
  File "/usr/lib/python3.4/runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.4/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/root/.local/lib/python3.4/site-packages/upip.py", line 315, in <module>
    main()
  File "/root/.local/lib/python3.4/site-packages/upip.py", line 308, in main
    install(to_install)
  File "/root/.local/lib/python3.4/site-packages/upip.py", line 230, in install
    raise e
  File "/root/.local/lib/python3.4/site-packages/upip.py", line 218, in install
    meta = install_pkg(pkg_spec, install_path)
  File "/root/.local/lib/python3.4/site-packages/upip.py", line 173, in install_pkg
    data = get_pkg_metadata(pkg_spec)
  File "/root/.local/lib/python3.4/site-packages/upip.py", line 161, in get_pkg_metadata
    return json.load(f)
  File "/usr/lib/python3.4/json/__init__.py", line 268, in load
    parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
  File "/usr/lib/python3.4/json/__init__.py", line 312, in loads
    s.__class__.__name__))
TypeError: the JSON object must be str, not 'bytes'

[pfalcon]

So, that shows how to do it in a way that it furthers the state of affairs and opens new possibilities, not works around things, adds maintenance burden, and arguably splits community.

It was done over the course of watching 40-min documentary (which turned it into like 2-hr watching, flaming modern multitasking humans).

The point is: of course, it's not "complete" and going to have issues. But like any other issues, they are there to resolve, to again further the state of affairs vs being waste of time.

The above shows the apparent difference in CPython 3.6 I have here and your CPython 3.4. And in general, that's quite a problem, because they break compatibility like mad dogs. Here however it's not the case, the issue should be trivial, please investigate it and patches are welcome at http://github.com/pfalcon/micropython-lib .

[pfalcon]

Btw this:

  File "/root/.local/lib/python3.4/site-packages/upip.py", line 161, in get_pkg_metadata
    return json.load(f)
  File "/usr/lib/python3.4/json/__init__.py", line 268, in load

leads to concerns that what you have as /root/.local/lib/python3.4/site-packages/upip.py is not what was installed as pip3 install --user micropython-cpython-upip.

[pfalcon]

leads to concerns

Ah, nope, ignore that. That's indeed how from json import load would affect the stacktrace (i.e., in no way), though that may be not obvious immediately. Only proves the point that there's always something to learn and speculative debugging is bound to lead to goofs, so one should be prepared to exactly investigate an issue on their side, not relying on "telepathy" ;-).

[peterhinch]

This is evidently contentious. I am therefore closing this PR.