New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider ways to efficiently implement support for multiple platforms #9

Open
pfalcon opened this Issue Oct 22, 2014 · 11 comments

Comments

Projects
None yet
4 participants
@pfalcon
Member

pfalcon commented Oct 22, 2014

There's growing interest to support multiple platforms in micropython-lib besides currently supported POSIX/Linux - MacOSX #7, bare-metal (PyBoard) #8, Windows micropython/micropython#886 (comment). Consider ways how to implement it efficiently.

@dpgeorge

This comment has been minimized.

Show comment
Hide comment
@dpgeorge

dpgeorge Oct 22, 2014

Contributor

Yes.

If we have this feature then we should go just all out and use micropython-lib for all libraries, including hardware modules (eg NRF24L01 driver currently living in micropython/drivers could move here).

Contributor

dpgeorge commented Oct 22, 2014

Yes.

If we have this feature then we should go just all out and use micropython-lib for all libraries, including hardware modules (eg NRF24L01 driver currently living in micropython/drivers could move here).

@turbinenreiter

This comment has been minimized.

Show comment
Hide comment
@turbinenreiter

turbinenreiter Oct 26, 2014

I cant believe there has been a NRF24L01 driver for almost a month and I didn't see it.
I think it's a good idea to move hardware modules here - to avoid just that.

Maybe we can get pip to do the work for us. From a user perspective, I would like to do:
micropython-pip install -t pyboard micropython-module
or maybe:
micropython-pip deploy micropython-module

Every module would need to have a note about the platform, maybe in setup.py, i.E.
platform=[unix, pyboard]

The question is if it's possible to get pip to do that, without breaking it (we had something similar already, but it was crap because it 'stole' flags from pip) and without having to set environment variables (like it's done now), because that sucks.

turbinenreiter commented Oct 26, 2014

I cant believe there has been a NRF24L01 driver for almost a month and I didn't see it.
I think it's a good idea to move hardware modules here - to avoid just that.

Maybe we can get pip to do the work for us. From a user perspective, I would like to do:
micropython-pip install -t pyboard micropython-module
or maybe:
micropython-pip deploy micropython-module

Every module would need to have a note about the platform, maybe in setup.py, i.E.
platform=[unix, pyboard]

The question is if it's possible to get pip to do that, without breaking it (we had something similar already, but it was crap because it 'stole' flags from pip) and without having to set environment variables (like it's done now), because that sucks.

@dpgeorge

This comment has been minimized.

Show comment
Hide comment
@dpgeorge

dpgeorge Oct 26, 2014

Contributor

I cant believe there has been a NRF24L01 driver for almost a month and I didn't see it.

Well, these things need better advertising :) Like making documentation for the modules available at docs.micropython.org.

Every module would need to have a note about the platform, maybe in setup.py

But some modules require different implementations for different platforms. Eg on unix ffi is used a lot, but this can't be done on pyboard. And we don't want to have code for all platforms in the 1 file, with run-time autodetection, because that's bloat.

Contributor

dpgeorge commented Oct 26, 2014

I cant believe there has been a NRF24L01 driver for almost a month and I didn't see it.

Well, these things need better advertising :) Like making documentation for the modules available at docs.micropython.org.

Every module would need to have a note about the platform, maybe in setup.py

But some modules require different implementations for different platforms. Eg on unix ffi is used a lot, but this can't be done on pyboard. And we don't want to have code for all platforms in the 1 file, with run-time autodetection, because that's bloat.

@pfalcon

This comment has been minimized.

Show comment
Hide comment
@pfalcon

pfalcon Oct 26, 2014

Member

And we don't want to have code for all platforms in the 1 file, with run-time autodetection, because that's bloat.

Yes, that's exactly what I meant. Let's have some examples:

Naive way to implement multi-platform support is (taking select module as an example):

(1)

if sys.platform == "linux":
    def select():
        ...
elif sys.platform == "windows":
    def select():
        ...
elif sys.platform == "pyboard":
    def select():
        ...

But that will waste memory to store bytecode for all platform, even though only one of them will be used at a time.

That can be improved with:

(2)

if sys.platform == "linux":
    from select_linux import *
elif sys.platform == "windows":
    from select_windows import *
elif sys.platform == "pyboard":
    from select_pyboard import *

That won't waste as much memory, but kinda assumed that all of select_linux.py, select_windows.py, select_pyboard.py are shipped and installed with "select" module. But that will waste storage space in "file system".

Can we improve from that? Can we ship submodules for all platforms in PyPI package, but install only one needed for target platform? That kinda can be achieved even with (2), by having good naming conventions and post-processing step to remove modules unneeded for target platform.

What if go even bit further and making sure that "select.py" module installed is platform-specific code, without extra redirection step like in (2). Does standard Python tools (setuptools, etc.) support anything like that?

And with all that, we would need support for "cross-installs" - like, prepare on Linux library snapshot for PyBoard target.

Member

pfalcon commented Oct 26, 2014

And we don't want to have code for all platforms in the 1 file, with run-time autodetection, because that's bloat.

Yes, that's exactly what I meant. Let's have some examples:

Naive way to implement multi-platform support is (taking select module as an example):

(1)

if sys.platform == "linux":
    def select():
        ...
elif sys.platform == "windows":
    def select():
        ...
elif sys.platform == "pyboard":
    def select():
        ...

But that will waste memory to store bytecode for all platform, even though only one of them will be used at a time.

That can be improved with:

(2)

if sys.platform == "linux":
    from select_linux import *
elif sys.platform == "windows":
    from select_windows import *
elif sys.platform == "pyboard":
    from select_pyboard import *

That won't waste as much memory, but kinda assumed that all of select_linux.py, select_windows.py, select_pyboard.py are shipped and installed with "select" module. But that will waste storage space in "file system".

Can we improve from that? Can we ship submodules for all platforms in PyPI package, but install only one needed for target platform? That kinda can be achieved even with (2), by having good naming conventions and post-processing step to remove modules unneeded for target platform.

What if go even bit further and making sure that "select.py" module installed is platform-specific code, without extra redirection step like in (2). Does standard Python tools (setuptools, etc.) support anything like that?

And with all that, we would need support for "cross-installs" - like, prepare on Linux library snapshot for PyBoard target.

@dpgeorge

This comment has been minimized.

Show comment
Hide comment
@dpgeorge

dpgeorge Oct 26, 2014

Contributor

What if go even bit further and making sure that "select.py" module installed is platform-specific code,

Yes, I'd like to see it go this way. I think it's the only solution worth working on, even if it has lots of hurdles.

Contributor

dpgeorge commented Oct 26, 2014

What if go even bit further and making sure that "select.py" module installed is platform-specific code,

Yes, I'd like to see it go this way. I think it's the only solution worth working on, even if it has lots of hurdles.

@blmorris

This comment has been minimized.

Show comment
Hide comment
@blmorris

blmorris Oct 30, 2014

Contributor

Today I started trying to figure out how to use the uasyncio library on the pyboard - not an immediately pressing need, but I would like to learn how to use this cooperative multitasking model for some of my projects and this seemed like a good place to start.
I immediately ran into problems and did not get very far. Some of it is just that I am still on the steep part of the learning curve for all things Python and don't always know where to start, and can't tell if the problem I am having is an actual bug or just user error. Some of it is that the OSX versions of 'find' (used in the micropython-lib Makefile) and 'install' (used in the 'make install' process for the unix port) use different options than their Linux counterparts.
Eventually i got around enough of these problems to try to use pip-micropython to install the uasyncio library for the unix port

unix bryan$ pip-micropython install micropython-uasyncio
Warning: MICROPYPATH is not set, assuming default value
Destination library directory: /Users/bryan/.micropython/lib
Downloading/unpacking micropython-uasyncio
  Downloading micropython-uasyncio-0.7.tar.gz
  Running setup.py (path:/private/var/folders/hr/_blbpnt10419kd3gkq7z2mgh0000gn/T/pip-micropy-venv/build/micropython-uasyncio/setup.py) egg_info for package micropython-uasyncio

...lots more messages, then this error:
Downloading/unpacking micropython-select (from micropython-uasyncio)
  Downloading micropython-select-0.1.tar.gz
  Running setup.py (path:/private/var/folders/hr/_blbpnt10419kd3gkq7z2mgh0000gn/T/pip-micropy-venv/build/micropython-select/setup.py) egg_info for package micropython-select
    Traceback (most recent call last):
      File "<string>", line 3, in <module>
      File "/var/folders/hr/_blbpnt10419kd3gkq7z2mgh0000gn/T/pip-micropy-venv/lib/python3.4/site-packages/setuptools/__init__.py", line 12, in <module>
        from setuptools.extension import Extension
      File "/var/folders/hr/_blbpnt10419kd3gkq7z2mgh0000gn/T/pip-micropy-venv/lib/python3.4/site-packages/setuptools/extension.py", line 7, in <module>
        from setuptools.dist import _get_unpatched
      File "/var/folders/hr/_blbpnt10419kd3gkq7z2mgh0000gn/T/pip-micropy-venv/lib/python3.4/site-packages/setuptools/dist.py", line 16, in <module>
        from setuptools.compat import numeric_types, basestring
      File "/var/folders/hr/_blbpnt10419kd3gkq7z2mgh0000gn/T/pip-micropy-venv/lib/python3.4/site-packages/setuptools/compat.py", line 49, in <module>
        from http.server import HTTPServer, SimpleHTTPRequestHandler
      File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/server.py", line 93, in <module>
        import select
      File "/private/var/folders/hr/_blbpnt10419kd3gkq7z2mgh0000gn/T/pip-micropy-venv/build/micropython-select/select.py", line 1, in <module>
        import ffi
    ImportError: No module named 'ffi'

This surprised me, as I have libffi installed, and can use 'import ffi' in micropython, but as far as I can tell CPython has a 'cffi' rather than 'ffi' module, and somehow select.py was being executed by my python3 installation and not by micropython. I couldn't tell if this is was was supposed to happen, or how it is really supposed to work.
All these efforts came after I tried to install uasyncio.core on my pyboard and start writing code for it; it didn't seem to do what I thought it should (maybe this is the testing that it needs?) but I want to make sure that I am at least starting in a reasonable configuration before I start posting bug reports or trying to fix something that is only broken because I didn't know how to install it. I started with the coroutine examples here - https://docs.python.org/3.4/library/asyncio-task.html - and used import core as asyncio to run the samples. Some parts worked, but few of my attempts to extend the examples did; I just want to make sure I'm not doing anything obviously wrong before going further with it.
-Bryan

Contributor

blmorris commented Oct 30, 2014

Today I started trying to figure out how to use the uasyncio library on the pyboard - not an immediately pressing need, but I would like to learn how to use this cooperative multitasking model for some of my projects and this seemed like a good place to start.
I immediately ran into problems and did not get very far. Some of it is just that I am still on the steep part of the learning curve for all things Python and don't always know where to start, and can't tell if the problem I am having is an actual bug or just user error. Some of it is that the OSX versions of 'find' (used in the micropython-lib Makefile) and 'install' (used in the 'make install' process for the unix port) use different options than their Linux counterparts.
Eventually i got around enough of these problems to try to use pip-micropython to install the uasyncio library for the unix port

unix bryan$ pip-micropython install micropython-uasyncio
Warning: MICROPYPATH is not set, assuming default value
Destination library directory: /Users/bryan/.micropython/lib
Downloading/unpacking micropython-uasyncio
  Downloading micropython-uasyncio-0.7.tar.gz
  Running setup.py (path:/private/var/folders/hr/_blbpnt10419kd3gkq7z2mgh0000gn/T/pip-micropy-venv/build/micropython-uasyncio/setup.py) egg_info for package micropython-uasyncio

...lots more messages, then this error:
Downloading/unpacking micropython-select (from micropython-uasyncio)
  Downloading micropython-select-0.1.tar.gz
  Running setup.py (path:/private/var/folders/hr/_blbpnt10419kd3gkq7z2mgh0000gn/T/pip-micropy-venv/build/micropython-select/setup.py) egg_info for package micropython-select
    Traceback (most recent call last):
      File "<string>", line 3, in <module>
      File "/var/folders/hr/_blbpnt10419kd3gkq7z2mgh0000gn/T/pip-micropy-venv/lib/python3.4/site-packages/setuptools/__init__.py", line 12, in <module>
        from setuptools.extension import Extension
      File "/var/folders/hr/_blbpnt10419kd3gkq7z2mgh0000gn/T/pip-micropy-venv/lib/python3.4/site-packages/setuptools/extension.py", line 7, in <module>
        from setuptools.dist import _get_unpatched
      File "/var/folders/hr/_blbpnt10419kd3gkq7z2mgh0000gn/T/pip-micropy-venv/lib/python3.4/site-packages/setuptools/dist.py", line 16, in <module>
        from setuptools.compat import numeric_types, basestring
      File "/var/folders/hr/_blbpnt10419kd3gkq7z2mgh0000gn/T/pip-micropy-venv/lib/python3.4/site-packages/setuptools/compat.py", line 49, in <module>
        from http.server import HTTPServer, SimpleHTTPRequestHandler
      File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/server.py", line 93, in <module>
        import select
      File "/private/var/folders/hr/_blbpnt10419kd3gkq7z2mgh0000gn/T/pip-micropy-venv/build/micropython-select/select.py", line 1, in <module>
        import ffi
    ImportError: No module named 'ffi'

This surprised me, as I have libffi installed, and can use 'import ffi' in micropython, but as far as I can tell CPython has a 'cffi' rather than 'ffi' module, and somehow select.py was being executed by my python3 installation and not by micropython. I couldn't tell if this is was was supposed to happen, or how it is really supposed to work.
All these efforts came after I tried to install uasyncio.core on my pyboard and start writing code for it; it didn't seem to do what I thought it should (maybe this is the testing that it needs?) but I want to make sure that I am at least starting in a reasonable configuration before I start posting bug reports or trying to fix something that is only broken because I didn't know how to install it. I started with the coroutine examples here - https://docs.python.org/3.4/library/asyncio-task.html - and used import core as asyncio to run the samples. Some parts worked, but few of my attempts to extend the examples did; I just want to make sure I'm not doing anything obviously wrong before going further with it.
-Bryan

@blmorris

This comment has been minimized.

Show comment
Hide comment
@blmorris

blmorris Oct 30, 2014

Contributor

Sorry if that last message rambled a bit off topic - I just decided to start working with the libraries yesterday, and immediately got stuck in several different ways. I got to the point where I couldn't really tell which problems were with my own system, which ones were my own mistakes, and which ones were bugs in the libraries that need to be fixed. I figured that instead of hacking on kludgy workarounds that would just break things further, I should just stop and write up the problems I had as well as I could.

Contributor

blmorris commented Oct 30, 2014

Sorry if that last message rambled a bit off topic - I just decided to start working with the libraries yesterday, and immediately got stuck in several different ways. I got to the point where I couldn't really tell which problems were with my own system, which ones were my own mistakes, and which ones were bugs in the libraries that need to be fixed. I figured that instead of hacking on kludgy workarounds that would just break things further, I should just stop and write up the problems I had as well as I could.

@pfalcon

This comment has been minimized.

Show comment
Hide comment
@pfalcon

pfalcon Oct 30, 2014

Member

@blmorris: Yes, this isn't exactly the right ticket for this. There's a topic on forum for uasyncio: http://forum.micropython.org/viewtopic.php?f=3&t=85 . I'll reply to some points here so they don't go unanswered, but please move further discussion to the forum (or separate topics). Thanks.

Some of it is that the OSX versions of 'find' (used in the micropython-lib Makefile) and 'install' (used in the 'make install' process for the unix port) use different options than their Linux counterparts.

Please report these (with enough details) to https://github.com/micropython/micropython/issues/new

ImportError: No module named 'ffi'

This is micropython/micropython#839 . @dpgeorge experienced it on ArchLinux. As it says, I don't know why it happens for some Python versions, even despite there're workarounds specifically against that. Current plan is not to workaround pip further, but to write lightweight install (upip?) specifically for uPy.

All these efforts came after I tried to install uasyncio.core

Yes, as http://forum.micropython.org/viewtopic.php?f=3&t=85&start=10#p1952 says, only uasyncio.core is intended to run on pyboard, full uasyncio won't.

uasyncio.core on my pyboard and start writing code for it; it didn't seem to do what I thought it should

Feel free to share your troubles in the forum topic. Async programming is not the easiest topic, and even with "yield from" which makes control flow more visible, there're still lot's of caveats. For example, if you forget "yield from" somewhere, you're hosed - it doesn't work, and you usually don't get any error. CPython asyncio has some debugging measures for such cases, which we won't afford, so it's important to stay compatible (even if in micropython/micropython#919 way), which is still way to go.

import core as asyncio

Ideally it should be "import uasyncio.core as asyncio" - do it right from the start, then there will be less confusion later whether that's issue on your side or genuine bug ;-).

Member

pfalcon commented Oct 30, 2014

@blmorris: Yes, this isn't exactly the right ticket for this. There's a topic on forum for uasyncio: http://forum.micropython.org/viewtopic.php?f=3&t=85 . I'll reply to some points here so they don't go unanswered, but please move further discussion to the forum (or separate topics). Thanks.

Some of it is that the OSX versions of 'find' (used in the micropython-lib Makefile) and 'install' (used in the 'make install' process for the unix port) use different options than their Linux counterparts.

Please report these (with enough details) to https://github.com/micropython/micropython/issues/new

ImportError: No module named 'ffi'

This is micropython/micropython#839 . @dpgeorge experienced it on ArchLinux. As it says, I don't know why it happens for some Python versions, even despite there're workarounds specifically against that. Current plan is not to workaround pip further, but to write lightweight install (upip?) specifically for uPy.

All these efforts came after I tried to install uasyncio.core

Yes, as http://forum.micropython.org/viewtopic.php?f=3&t=85&start=10#p1952 says, only uasyncio.core is intended to run on pyboard, full uasyncio won't.

uasyncio.core on my pyboard and start writing code for it; it didn't seem to do what I thought it should

Feel free to share your troubles in the forum topic. Async programming is not the easiest topic, and even with "yield from" which makes control flow more visible, there're still lot's of caveats. For example, if you forget "yield from" somewhere, you're hosed - it doesn't work, and you usually don't get any error. CPython asyncio has some debugging measures for such cases, which we won't afford, so it's important to stay compatible (even if in micropython/micropython#919 way), which is still way to go.

import core as asyncio

Ideally it should be "import uasyncio.core as asyncio" - do it right from the start, then there will be less confusion later whether that's issue on your side or genuine bug ;-).

@pfalcon

This comment has been minimized.

Show comment
Hide comment
@pfalcon

pfalcon Oct 30, 2014

Member

Back to topic:

What if go even bit further and making sure that "select.py" module installed is platform-specific code,

Yes, I'd like to see it go this way. I think it's the only solution worth working on, even if it has lots of hurdles.

Thanks for confirming that. I also would prefer that, but I wasn't sure if I'm just being too perfectionistic ;-).

Then would be nice to figure if setuptools and friends offer such functionality already (like, best practices how to do that). I saw related topic on distutils list and sneezed chance to start the discussion (but didn't have chance to follow up much yet):

https://mail.python.org/pipermail/distutils-sig/2014-October/025145.html

Member

pfalcon commented Oct 30, 2014

Back to topic:

What if go even bit further and making sure that "select.py" module installed is platform-specific code,

Yes, I'd like to see it go this way. I think it's the only solution worth working on, even if it has lots of hurdles.

Thanks for confirming that. I also would prefer that, but I wasn't sure if I'm just being too perfectionistic ;-).

Then would be nice to figure if setuptools and friends offer such functionality already (like, best practices how to do that). I saw related topic on distutils list and sneezed chance to start the discussion (but didn't have chance to follow up much yet):

https://mail.python.org/pipermail/distutils-sig/2014-October/025145.html

@blmorris

This comment has been minimized.

Show comment
Hide comment
@blmorris

blmorris Oct 30, 2014

Contributor

@pfalcon - Thanks for addressing my questions; I realize it wasn't exactly the right venue, but I was having one of those "whatever you touch is broken" days (h/t micropython/micropython#912) and needed to dump a bunch of questions for a sanity check (especially the "No module named 'ffi'" issue, which I missed earlier). I'll collect some more data and report the issues with 'find' and 'install' and bring my asyncio questions over to the forum.

Contributor

blmorris commented Oct 30, 2014

@pfalcon - Thanks for addressing my questions; I realize it wasn't exactly the right venue, but I was having one of those "whatever you touch is broken" days (h/t micropython/micropython#912) and needed to dump a bunch of questions for a sanity check (especially the "No module named 'ffi'" issue, which I missed earlier). I'll collect some more data and report the issues with 'find' and 'install' and bring my asyncio questions over to the forum.

@dpgeorge

This comment has been minimized.

Show comment
Hide comment
@dpgeorge

dpgeorge Nov 2, 2014

Contributor

I wasn't sure if I'm just being too perfectionistic ;-).

No such thing! :)

Contributor

dpgeorge commented Nov 2, 2014

I wasn't sure if I'm just being too perfectionistic ;-).

No such thing! :)

@pfalcon pfalcon changed the title from Consider ways to efficiently implement support for multiple platform to Consider ways to efficiently implement support for multiple platforms Jan 1, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment