Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TypeError while generating stubs from board via USB #400

Closed
trepidacious opened this issue Mar 11, 2023 · 15 comments
Closed

TypeError while generating stubs from board via USB #400

trepidacious opened this issue Mar 11, 2023 · 15 comments
Labels
bug Something isn't working

Comments

@trepidacious
Copy link

Describe the bug

micropy fails to generate stubs from a PicoW board running Pimoroni Micropython, when no other stubs exist.

Expected Behavior

Micropy generates stubs successfully.

Current Behavior

Micropy produces a TypeError and doesn't generate the stubs.

Steps to Reproduce

  1. Make sure you have no stubs installed - e.g. rm -rf ~/.micropy
  2. Connect a Raspberry Pi Pico W board via USB, holding bootsel
  3. Copy the Pimoroni Micropython 1.19.16 image for Pico W to the board, from Releases page
  4. Run micropy via USB, with something like micropy stubs create /dev/cu.usbmodem11201

Possible Solution

For me, adding str around name in the line "/".join([package.repo_name, name]) in resolve_package function of repo.py fixed the issue.

    def resolve_package(self, name: str) -> StubRepositoryPackage:
        """Resolve a package name to a package path.

        Args:
            name: Package name.

        Returns:
            Package location.

        Throws:
            StubNotFound: When package cannot be resolved.

        """
        for package in self.search(str(name)):
            if package.match_exact(name) or package.match_exact(
                "/".join([package.repo_name, str(name)])
            ):
                return package
            latest = self.latest_for_package(package)
            if latest and latest.name == name:
                return latest
        raise exc.StubNotFound(name)

Once the stubs have generated once with this change, the unmodified code works fine - seems to be something about having no stubs, or maybe not having an existing version of the same-named stubs?

Logs

The last section of output from micropy stubs create /dev/cu.usbmodem11201 - everything before this looks normal:

MicroPy  Pyboard  Reading /stubs/micropython-rp2-1_19_1/websocket.py: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| [329/329 @ 320B/s]
MicroPy  Pyboard  Reading /stubs/micropython-rp2-1_19_1/zlib.py: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| [273/273 @ 269B/s]
MicroPy  Copied Stubs: stubs
Traceback (most recent call last):
  File "/opt/homebrew/bin/micropy", line 8, in <module>
    sys.exit(cli())
  File "/opt/homebrew/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/opt/homebrew/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/opt/homebrew/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/homebrew/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/homebrew/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/homebrew/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/opt/homebrew/lib/python3.10/site-packages/click/decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/opt/homebrew/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/opt/homebrew/lib/python3.10/site-packages/micropy/cli.py", line 306, in create
    return mpy.create_stubs(port, verbose=verbose)
  File "/opt/homebrew/lib/python3.10/site-packages/micropy/main.py", line 143, in create_stubs
    stub = self.stubs.add(stub_path)
  File "/opt/homebrew/lib/python3.10/site-packages/micropy/stubs/stubs.py", line 336, in add
    return self._load(stub_source, copy_to=dest)
  File "/opt/homebrew/lib/python3.10/site-packages/micropy/stubs/stubs.py", line 98, in _load
    with stub_source.ready() as src_path:
  File "/opt/homebrew/Cellar/python@3.10/3.10.6_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/opt/homebrew/lib/python3.10/site-packages/micropy/stubs/source.py", line 77, in ready
    path = reduce(reducer, self.locators, location or self.location)
  File "/opt/homebrew/lib/python3.10/site-packages/micropy/stubs/source.py", line 52, in _do_locate
    response = locator.prepare(path)
  File "/opt/homebrew/lib/python3.10/site-packages/micropy/stubs/source.py", line 138, in prepare
    source = self.repo.resolve_package(location)
  File "/opt/homebrew/lib/python3.10/site-packages/micropy/stubs/repo.py", line 134, in resolve_package
    "/".join([package.repo_name, name])
TypeError: sequence item 1: expected str instance, PosixPath found

micropy.log

Context (Environment)

  • OS: macOS Ventura 13.2.1, M1 processor
  • Micropy Version: 4.1.0
  • Python Version: Python 3.10.6
  • VSCode Version: 1.76.0
@trepidacious trepidacious added the bug Something isn't working label Mar 11, 2023
@BradenM
Copy link
Owner

BradenM commented Mar 11, 2023

@trepidacious
Thanks for the detailed report 😄

I will take a deeper look into this this weekend, but one thing I'm curious about is whether you see the same behavior when using the rshell backend (via micropy stubs create --backend rshell ...)

That would help narrow down where the root of the issue is (stub repo/management or pyb with pydevice backend, otherwise).

@trepidacious
Copy link
Author

@BradenM You're very welcome, thanks for the really useful tool! :)

I've tried running with that option but I get an error, not sure whether I'm using the right params?

$micropy stubs create --backend rshell /dev/cu.usbmodem11201 
Usage: micropy stubs create [OPTIONS] PORT
Try 'micropy stubs create --help' for help.

Error: No such option: --backend

I can only see --help and --verbose options when I run micropy stubs create --help.

rshell is installed and works with the board on the same serial device.

One thing I forgot to mention - following the docs for creating stubs I ran pip install micropy-cli[create_stubs] (actually I had to use pip install "micropy-cli[create_stubs]" on zsh), but this gave:

Requirement already satisfied: micropy-cli[create_stubs] in /opt/homebrew/lib/python3.10/site-packages (4.1.0)
WARNING: micropy-cli 4.1.0 does not provide the extra 'create_stubs'
...

Not sure whether that means I'm missing a dependency?

@BradenM
Copy link
Owner

BradenM commented Mar 13, 2023

@trepidacious
🤦 I seem to have gotten a bit ahead of myself… the functionality I was referring to is not released yet and is still in a PR #394

🙃 Sorry about that.

That said, it looks as if I may have already solved this issue in #394:
https://github.com/BradenM/micropy-cli/pull/394/files#diff-defa6605073200674e421571ce0f08acdde68675dd1c5e97237c360073c3b741L143-R152

I intend on pushing out a prelease once I get it merged (or at most soon after). I will leave this issue open for now so others may find your workaround if needed.

@BradenM
Copy link
Owner

BradenM commented Mar 13, 2023

One thing I forgot to mention - following the docs for creating stubs I ran pip install micropy-cli[create_stubs] (actually I had to use pip install "micropy-cli[create_stubs]" on zsh), but this gave

This is expected. The create_stubs extra was removed a while back -- I have just been slow to update the documentation 😬 (#395)

@trepidacious
Copy link
Author

@BradenM Ah that's great - I'll stand by for the prerelease, thanks!

@BradenM
Copy link
Owner

BradenM commented Mar 20, 2023

@trepidacious
https://github.com/BradenM/micropy-cli/releases/tag/v4.2.0-beta is out -- please let me know if you find the time to test it out and can confirm this issue can be closed Thanks!

@trepidacious
Copy link
Author

@BradenM Thanks, I've had a look and unfortunately I still get an error, it seems to be different now though:

MicroPy  Pyboard  Reading /stubs/micropython-v1_19_1-rp2/network.py: |                                                                                                         | [0.00/0.00 @ ?B/s]
MicroPy  Pyboard  Reading /stubs/micropython-v1_19_1-rp2/ntptime.py: |                                                                                                         | [0.00/0.00 @ ?B/s]
MicroPy  Pyboard  Reading /stubs/micropython-v1_19_1-rp2/onewire.py: |                                                                                                         | [0.00/0.00 @ ?B/s]
MicroPy  Copied Stubs: stubs
Traceback (most recent call last):
  File "/opt/homebrew/bin/micropy", line 8, in <module>
    sys.exit(cli())
  File "/opt/homebrew/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/opt/homebrew/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/opt/homebrew/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/homebrew/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/homebrew/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/homebrew/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/opt/homebrew/lib/python3.10/site-packages/click/decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/opt/homebrew/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/opt/homebrew/lib/python3.10/site-packages/micropy/cli.py", line 314, in create
    return mpy.create_stubs(port, verbose=verbose, backend=backend)
  File "/opt/homebrew/lib/python3.10/site-packages/micropy/main.py", line 151, in create_stubs
    stub_path = self.stubs.from_stubber(stub_path, out_dir)
  File "/opt/homebrew/lib/python3.10/site-packages/micropy/stubs/stubs.py", line 355, in from_stubber
    mod_file = next(_path.rglob("modules.json"))
StopIteration

I used pip install --upgrade --pre micropy-cli to upgrade, micropy --version gives micropy, version 4.2.0b0

I've repeated a few times with the same results - it looks like nothing is produced in ~/.micropy/stubs

micropy.log shows:

[2023-03-22 20:05:55] DEBUG: hookproxy(add_from_file): Method added to proxy: (<function PackagesModule.add_from_file at 0x10569b7f0>, '_hook__add_from_file__path_None__dev_False')
[2023-03-22 20:05:55] DEBUG: hookproxy(add_package): Method added to proxy: (<function PackagesModule.add_package at 0x10569b910>, '_hook__add_package__dev_False')
[2023-03-22 20:05:55] DEBUG: hookproxy(add_package): Method added to proxy: (<function DevPackagesModule.add_package at 0x10569bd90>, '_hook__add_package__dev_True')
[2023-03-22 20:05:55] DEBUG: hookproxy(add_from_file): Method added to proxy: (<function DevPackagesModule.add_from_file at 0x10569beb0>, '_hook__add_from_file__path_None__dev_True')
[2023-03-22 20:05:55] DEBUG: hookproxy(stubs): Method added to proxy: (<function StubsModule.stubs at 0x107ec31c0>, '_hook__stubs__')
[2023-03-22 20:05:55] DEBUG: hookproxy(add_stub): Method added to proxy: (<function StubsModule.add_stub at 0x107ec3640>, '_hook__add_stub__')
[2023-03-22 20:05:55] DEBUG: micropy: MicroPy Loaded
[2023-03-22 20:05:55] DEBUG: micropy: Running first time setup...
[2023-03-22 20:05:55] DEBUG: micropy: Creating .micropy directory @ /Users/rebeam/.micropy
[2023-03-22 20:05:55] INFO: micropy: Connecting to Pyboard @ /dev/cu.usbmodem21201
[2023-03-22 20:05:55] DEBUG: pyboard: Starting new HTTPS connection (1): raw.githubusercontent.com:443
[2023-03-22 20:05:55] DEBUG: pyboard: Starting new HTTPS connection (1): raw.githubusercontent.com:443
[2023-03-22 20:05:55] DEBUG: pyboard: https://raw.githubusercontent.com:443 "GET /BradenM/micropy-stubs/master/source.json HTTP/1.1" 200 1535
[2023-03-22 20:05:55] DEBUG: pyboard: https://raw.githubusercontent.com:443 "GET /Josverl/micropython-stubs/main/publish/package_data.jsondb HTTP/1.1" 200 4053
[2023-03-22 20:05:56] INFO: micropy: Connected!
[2023-03-22 20:05:56] INFO: micropy: Executing stubber on pyboard...
[2023-03-22 20:06:05] INFO: micropy: Done!
[2023-03-22 20:06:05] INFO: micropy: Copying stubs...
[2023-03-22 20:08:50] INFO: micropy: Copied Stubs: stubs
[2023-03-23 09:00:26] DEBUG: hookproxy(add_from_file): Method added to proxy: (<function PackagesModule.add_from_file at 0x105c437f0>, '_hook__add_from_file__path_None__dev_False')
[2023-03-23 09:00:26] DEBUG: hookproxy(add_package): Method added to proxy: (<function PackagesModule.add_package at 0x105c43910>, '_hook__add_package__dev_False')
[2023-03-23 09:00:26] DEBUG: hookproxy(add_package): Method added to proxy: (<function DevPackagesModule.add_package at 0x105c43d90>, '_hook__add_package__dev_True')
[2023-03-23 09:00:26] DEBUG: hookproxy(add_from_file): Method added to proxy: (<function DevPackagesModule.add_from_file at 0x105c43eb0>, '_hook__add_from_file__path_None__dev_True')
[2023-03-23 09:00:26] DEBUG: hookproxy(stubs): Method added to proxy: (<function StubsModule.stubs at 0x10846b1c0>, '_hook__stubs__')
[2023-03-23 09:00:26] DEBUG: hookproxy(add_stub): Method added to proxy: (<function StubsModule.add_stub at 0x10846b640>, '_hook__add_stub__')

@trepidacious
Copy link
Author

Just realised I'd chopped off a bit too much - there's also a "Failed to create the report" message earlier in the output:

MicroPy  Pyboard  Stub module: urandom                   to file: /stubs/micropython-v1_19_1-rp2/urandom.py                              mem:116656
MicroPy  Pyboard  Stub module: ure                       to file: /stubs/micropython-v1_19_1-rp2/ure.py                                  mem:116656
MicroPy  Pyboard  Stub module: urequests                 to file: /stubs/micropython-v1_19_1-rp2/urequests.py                            mem:116656
MicroPy  Pyboard  Stub module: urllib/urequest           to file: /stubs/micropython-v1_19_1-rp2/urllib/urequest.py             Skip module: urllib.urequest           Module not found.
MicroPy  Pyboard  Stub module: uselect                   to file: /stubs/micropython-v1_19_1-rp2/uselect.py                              mem:116640
MicroPy  Pyboard  Stub module: usocket                   to file: /stubs/micropython-v1_19_1-rp2/usocket.py                              mem:116640
MicroPy  Pyboard  Stub module: ussl                      to file: /stubs/micropython-v1_19_1-rp2/ussl.py                                 mem:116640
MicroPy  Pyboard  Stub module: ustruct                   to file: /stubs/micropython-v1_19_1-rp2/ustruct.py                              mem:116640
MicroPy  Pyboard  Stub module: usys                      to file: /stubs/micropython-v1_19_1-rp2/usys.py                                 mem:116640
MicroPy  Pyboard  Stub module: utelnetserver             to file: /stubs/micropython-v1_19_1-rp2/utelnetserver.py               Skip module: utelnetserver             Module not found.
MicroPy  Pyboard  Stub module: utime                     to file: /stubs/micropython-v1_19_1-rp2/utime.py                                mem:116624
MicroPy  Pyboard  Stub module: utimeq                    to file: /stubs/micropython-v1_19_1-rp2/utimeq.py                      Skip module: utimeq                    Module not found.
MicroPy  Pyboard  Stub module: uwebsocket                to file: /stubs/micropython-v1_19_1-rp2/uwebsocket.py                           mem:116608
MicroPy  Pyboard  Stub module: uzlib                     to file: /stubs/micropython-v1_19_1-rp2/uzlib.py                                mem:116608
MicroPy  Pyboard  Stub module: version                   to file: /stubs/micropython-v1_19_1-rp2/version.py                              mem:116608
MicroPy  Pyboard  Stub module: websocket                 to file: /stubs/micropython-v1_19_1-rp2/websocket.py                            mem:116528
MicroPy  Pyboard  Stub module: websocket_helper          to file: /stubs/micropython-v1_19_1-rp2/websocket_helper.py            Skip module: websocket_helper          Module not found.
MicroPy  Pyboard  Stub module: wipy                      to file: /stubs/micropython-v1_19_1-rp2/wipy.py                        Skip module: wipy                      Module not found.
MicroPy  Pyboard  Stub module: writer                    to file: /stubs/micropython-v1_19_1-rp2/writer.py                      Skip module: writer                    Module not found.
MicroPy  Pyboard  Stub module: xpt2046                   to file: /stubs/micropython-v1_19_1-rp2/xpt2046.py                     Skip module: xpt2046                   Module not found.
MicroPy  Pyboard  Stub module: ymodem                    to file: /stubs/micropython-v1_19_1-rp2/ymodem.py                      Skip module: ymodem                    Module not found.
MicroPy  Pyboard  Stub module: zephyr                    to file: /stubs/micropython-v1_19_1-rp2/zephyr.py                      Skip module: zephyr                    Module not found.
MicroPy  Pyboard  Stub module: zlib                      to file: /stubs/micropython-v1_19_1-rp2/zlib.py                                 mem:116464
MicroPy  Pyboard  Created stubs for 48 modules on board micropython-v1.19.1-rp2
MicroPy  Pyboard  Path: /stubs/micropython-v1_19_1-rp2
MicroPy  Pyboard  Failed to create the report.
MicroPy  ✔ Done!
MicroPy  Copying stubs...
MicroPy  Pyboard  Reading /stubs/micropython-v1_19_1-rp2/_boot_fat.py: 100%|██████████████| [172/172 @ 279B/s]
MicroPy  Pyboard  Reading /stubs/micropython-v1_19_1-rp2/_onewire.py: 100%|███████████████| [309/309 @ 302B/s]

I grabbed a copy of the files that had been copied to the temp directory at the point the script fails, interestingly the last few are 0 bytes, these are also the files that display "[0.00/0.00 @ ?B/s]" in the micropy output. The file names also stop before the end of the list above - there's nothing after onewire, even though there are other packages listed in the stub creation output from micropy (e.g. uzlib, version, websocket etc. Is there any chance it's running out of RAM/Flash on the board?

-rw-r--r--   1 rebeam  staff   344B Mar 23 10:33 _boot_fat.py
-rw-r--r--   1 rebeam  staff   618B Mar 23 10:33 _onewire.py
-rw-r--r--   1 rebeam  staff   1.9K Mar 23 10:33 _rp2.py
-rw-r--r--   1 rebeam  staff   849B Mar 23 10:33 _thread.py
-rw-r--r--   1 rebeam  staff   732B Mar 23 10:33 _uasyncio.py
-rw-r--r--   1 rebeam  staff   542B Mar 23 10:33 adcfft.py
-rw-r--r--   1 rebeam  staff   536B Mar 23 10:33 array.py
-rw-r--r--   1 rebeam  staff   579B Mar 23 10:33 binascii.py
-rw-r--r--   1 rebeam  staff   1.6K Mar 23 10:33 breakout_as7262.py
-rw-r--r--   1 rebeam  staff   1.0K Mar 23 10:33 breakout_bh1745.py
-rw-r--r--   1 rebeam  staff   1.2K Mar 23 10:33 breakout_bme280.py
-rw-r--r--   1 rebeam  staff   1.4K Mar 23 10:33 breakout_bme68x.py
-rw-r--r--   1 rebeam  staff   1.3K Mar 23 10:34 breakout_bmp280.py
-rw-r--r--   1 rebeam  staff   933B Mar 23 10:34 breakout_dotmatrix.py
-rw-r--r--   1 rebeam  staff   1.1K Mar 23 10:34 breakout_encoder.py
-rw-r--r--   1 rebeam  staff   769B Mar 23 10:34 breakout_icp10125.py
-rw-r--r--   1 rebeam  staff   2.0K Mar 23 10:34 breakout_ioexpander.py
-rw-r--r--   1 rebeam  staff   1.5K Mar 23 10:34 breakout_ltr559.py
-rw-r--r--   1 rebeam  staff   678B Mar 23 10:34 breakout_matrix11x7.py
-rw-r--r--   1 rebeam  staff   1.5K Mar 23 10:34 breakout_mics6814.py
-rw-r--r--   1 rebeam  staff   2.5K Mar 23 10:34 breakout_msa301.py
-rw-r--r--   1 rebeam  staff   1.1K Mar 23 10:34 breakout_paa5100.py
-rw-r--r--   1 rebeam  staff   1.1K Mar 23 10:34 breakout_pmw3901.py
-rw-r--r--   1 rebeam  staff   1.1K Mar 23 10:34 breakout_potentiometer.py
-rw-r--r--   1 rebeam  staff   681B Mar 23 10:34 breakout_rgbmatrix5x5.py
-rw-r--r--   1 rebeam  staff   4.2K Mar 23 10:34 breakout_rtc.py
-rw-r--r--   1 rebeam  staff   808B Mar 23 10:34 breakout_scd41.py
-rw-r--r--   1 rebeam  staff   1.1K Mar 23 10:34 breakout_sgp30.py
-rw-r--r--   1 rebeam  staff   1.1K Mar 23 10:34 breakout_trackball.py
-rw-r--r--   1 rebeam  staff   1.7K Mar 23 10:34 breakout_vl53l5cx.py
-rw-r--r--   1 rebeam  staff   781B Mar 23 10:34 cmath.py
-rw-r--r--   1 rebeam  staff   1.3K Mar 23 10:34 collections.py
-rw-r--r--   1 rebeam  staff   540B Mar 23 10:35 cryptolib.py
-rw-r--r--   1 rebeam  staff   1.0K Mar 23 10:35 dht.py
-rw-r--r--   1 rebeam  staff   781B Mar 23 10:35 ds18x20.py
-rw-r--r--   1 rebeam  staff   1.3K Mar 23 10:35 encoder.py
-rw-r--r--   1 rebeam  staff   939B Mar 23 10:35 errno.py
-rw-r--r--   1 rebeam  staff   1.3K Mar 23 10:35 framebuf.py
-rw-r--r--   1 rebeam  staff   665B Mar 23 10:35 gc.py
-rw-r--r--   1 rebeam  staff   4.1K Mar 23 10:35 gfx_pack.py
-rw-r--r--   1 rebeam  staff   734B Mar 23 10:35 hashlib.py
-rw-r--r--   1 rebeam  staff   479B Mar 23 10:35 heapq.py
-rw-r--r--   1 rebeam  staff   1.2K Mar 23 10:35 hub75.py
-rw-r--r--   1 rebeam  staff   5.0K Mar 23 10:35 interstate75.py
-rw-r--r--   1 rebeam  staff   1.6K Mar 23 10:35 io.py
-rw-r--r--   1 rebeam  staff   859B Mar 23 10:35 jpegdec.py
-rw-r--r--   1 rebeam  staff   513B Mar 23 10:35 json.py
-rw-r--r--   1 rebeam  staff   1.9K Mar 23 10:35 lwip.py
-rw-r--r--   1 rebeam  staff    10K Mar 23 10:35 machine.py
-rw-r--r--   1 rebeam  staff     0B Mar 23 10:35 math.py
-rw-r--r--   1 rebeam  staff     0B Mar 23 10:36 micropython.py
-rw-r--r--   1 rebeam  staff     0B Mar 23 10:36 mip.py
-rw-r--r--   1 rebeam  staff     0B Mar 23 10:36 motor.py
-rw-r--r--   1 rebeam  staff     0B Mar 23 10:36 neopixel.py
-rw-r--r--   1 rebeam  staff     0B Mar 23 10:36 network.py
-rw-r--r--   1 rebeam  staff     0B Mar 23 10:36 ntptime.py
-rw-r--r--   1 rebeam  staff     0B Mar 23 10:36 onewire.py

@trepidacious
Copy link
Author

I've now tried again with a Pico Lipo with 16MB of flash, and that completes successfully - the Pico W only has 2MB of flash, and I believe some additional libraries. Of course it might be something other than the flash size causing the difference...

@BradenM
Copy link
Owner

BradenM commented Mar 27, 2023

@trepidacious
Thanks for the updates.

Your suspicion on the flash size is certainly plausible.

I've just pushed out https://github.com/BradenM/micropy-cli/releases/tag/v4.2.0-beta.1 https://github.com/BradenM/micropy-cli/releases/tag/v4.2.0-beta.2 (had a minor issue with v4.2.0.beta.1 release...) which exposes additional flags to the create command.

Namely, this includes --variant, which allows you to use a couple of different versions of micropython-stubber through the use of codemods (see: Josverl/micropython-stubber#298 for more details).

Although compiling w/ mpy-cross would be ideal here -- I have not yet implemented it. Meanwhile, I'm curious to see what results you achieve toying around with the variant option and the two provide backends. If you find the time, please let me know what you find.

Thanks!

@trepidacious
Copy link
Author

@BradenM

Hi, thanks for the update - I've repeated the test on a Pico W with each of the options for --variant ('base', 'mem', 'db', 'lvgl'), and unfortunately they all give the same error (looks to be the same in each case, and very similar to the error form the previous version):

MicroPy  Pyboard  Reading /stubs/micropython-v1_19_1-rp2/hub75.py: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| [633/633 @ 442B/s]
MicroPy  Pyboard  Reading /stubs/micropython-v1_19_1-rp2/interstate75.py: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| [2.52k/2.52k @ 547B/s]
MicroPy  Pyboard  Reading /stubs/micropython-v1_19_1-rp2/io.py: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| [838/838 @ 456B/s]
MicroPy  Pyboard  Reading /stubs/micropython-v1_19_1-rp2/jpegdec.py: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| [429/429 @ 420B/s]
MicroPy  Pyboard  Reading /stubs/micropython-v1_19_1-rp2/json.py: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| [256/256 @ 250B/s]
MicroPy  Pyboard  Reading /stubs/micropython-v1_19_1-rp2/lwip.py: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| [985/985 @ 538B/s]
MicroPy  Pyboard  Reading /stubs/micropython-v1_19_1-rp2/machine.py: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| [5.00k/5.00k @ 609B/s]
MicroPy  Pyboard  Reading /stubs/micropython-v1_19_1-rp2/math.py: |                                                                                                                                                                                                       | [0.00/0.00 @ ?B/s]
MicroPy  Pyboard  Reading /stubs/micropython-v1_19_1-rp2/micropython.py: |                                                                                                                                                                                                | [0.00/0.00 @ ?B/s]
MicroPy  Pyboard  Reading /stubs/micropython-v1_19_1-rp2/mip.py: |                                                                                                                                                                                                        | [0.00/0.00 @ ?B/s]
MicroPy  Pyboard  Reading /stubs/micropython-v1_19_1-rp2/motor.py: |                                                                                                                                                                                                      | [0.00/0.00 @ ?B/s]
MicroPy  Pyboard  Reading /stubs/micropython-v1_19_1-rp2/neopixel.py: |                                                                                                                                                                                                   | [0.00/0.00 @ ?B/s]
MicroPy  Pyboard  Reading /stubs/micropython-v1_19_1-rp2/network.py: |                                                                                                                                                                                                    | [0.00/0.00 @ ?B/s]
MicroPy  Pyboard  Reading /stubs/micropython-v1_19_1-rp2/ntptime.py: |                                                                                                                                                                                                    | [0.00/0.00 @ ?B/s]
MicroPy  Pyboard  Reading /stubs/micropython-v1_19_1-rp2/onewire.py: |                                                                                                                                                                                                    | [0.00/0.00 @ ?B/s]
MicroPy  Copied Stubs: stubs
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /opt/homebrew/lib/python3.10/site-packages/micropy/app/stubs.py:140 in stubs_create              │
│                                                                                                  │
│   137 │   │   out_dir = Path(tmpdir)                                                             │
│   138 │   │   stub_path = next(out_dir.iterdir())                                                │
│   139 │   │   log.info(f"Copied Stubs: $[{stub_path.name}]")                                     │
│ ❱ 140 │   │   stub_path = mp.stubs.from_stubber(stub_path, out_dir)                              │
│   141 │   │   stub = mp.stubs.add(str(stub_path))                                                │
│   142 │   pyb.disconnect()                                                                       │
│   143 │   log.success(f"Added {stub.name} to stubs!")                                            │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │       _get_desc = <function stubs_create.<locals>._get_desc at 0x1158d84c0>                  │ │
│ │         backend = <CreateBackend.upydevice: 'upydevice'>                                     │ │
│ │    create_stubs = <_io.StringIO object at 0x11595c550>                                       │ │
│ │             ctx = <click.core.Context object at 0x11592ceb0>                                 │ │
│ │         exclude = []                                                                         │ │
│ │             log = <micropy.logger.ServiceLog object at 0x11585b760>                          │ │
│ │ message_handler = MessageHandlers(                                                           │ │
│ │                   │   on_message=<function stubs_create.<locals>.<lambda> at 0x1158d85e0>,   │ │
│ │                   )                                                                          │ │
│ │          module = []                                                                         │ │
│ │              mp = <micropy.main.MicroPy object at 0x11585afb0>                               │ │
│ │         out_dir = PosixPath('/var/folders/vz/pbzdmz6x6xb87qnyslnwptwc0000gn/T/tmpt6pg1ojr')  │ │
│ │            port = '/dev/cu.usbmodem21201'                                                    │ │
│ │             pyb = <micropy.pyd.pydevice.PyDevice object at 0x11592d540>                      │ │
│ │         pyb_log = <micropy.logger.ServiceLog object at 0x11592d4e0>                          │ │
│ │       stub_path = PosixPath('/var/folders/vz/pbzdmz6x6xb87qnyslnwptwc0000gn/T/tmpt6pg1ojr/s… │ │
│ │          tmpdir = '/var/folders/vz/pbzdmz6x6xb87qnyslnwptwc0000gn/T/tmpt6pg1ojr'             │ │
│ │         variant = <CreateStubsVariant.BASE: 'base'>                                          │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│                                                                                                  │
│ /opt/homebrew/lib/python3.10/site-packages/micropy/stubs/stubs.py:355 in from_stubber            │
│                                                                                                  │
│   352 │   │   """                                                                                │
│   353 │   │   _path = Path(path).resolve()                                                       │
│   354 │   │   dest = Path(dest).resolve()                                                        │
│ ❱ 355 │   │   mod_file = next(_path.rglob("modules.json"))                                       │
│   356 │   │   path = mod_file.parent                                                             │
│   357 │   │   mod_data = json.load(mod_file.open())                                              │
│   358 │   │   dev_fware = mod_data["firmware"]                                                   │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │ _path = PosixPath('/private/var/folders/vz/pbzdmz6x6xb87qnyslnwptwc0000gn/T/tmpt6pg1ojr/stu… │ │
│ │  dest = PosixPath('/private/var/folders/vz/pbzdmz6x6xb87qnyslnwptwc0000gn/T/tmpt6pg1ojr')    │ │
│ │  path = PosixPath('/var/folders/vz/pbzdmz6x6xb87qnyslnwptwc0000gn/T/tmpt6pg1ojr/stubs')      │ │
│ │  self = <micropy.stubs.stubs.StubManager object at 0x1159fc610>                              │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
StopIteration

@BradenM
Copy link
Owner

BradenM commented Apr 1, 2023

@trepidacious Thank you for the update!

Looking at this now, it seems clear what is most likely occurring.

The pydevice backend has logic built into it that attempts to circumvent OOM errors by automatically rebooting the device and reducing the buffer size used for reading from the device.

However, this logic works with the expectation that reads (and/or pydevice) throw exceptions / errors during the read. What seems to be happening is it's just reading nothing.

So, while it remains true that compiling beforehand via mpy-cross would likely prevent (not solve) this issue, I will revisit this logic and implement improved checks to ensure the integrity of the files. This will also be needed for #312 (comment) and related issues (and its probaby something I should have done from the get go... 😬 )

Thank you again @trepidacious for your time in helping me iron these issues out. Much appreciated.

@BradenM
Copy link
Owner

BradenM commented Apr 19, 2023

@trepidacious Hi, finally found the time to get another release out.

Mpy cross compilation and file integrity verification are in, so this should be the one 🤞

@BradenM
Copy link
Owner

BradenM commented Apr 22, 2023

Going to go ahead and close this out. Please reach out to me if any issues arise and it needs to be opened, thanks!

@BradenM BradenM closed this as completed Apr 22, 2023
@marcusstenbeck
Copy link

I've been running into this for a few hours now. Here's my logs.

▶ micropy stubs create /dev/cu.usbmodem1301                        

MicroPy  Connecting to Pyboard @ /dev/cu.usbmodem1301
MicroPy  ✔ Connected!
2023-06-17 13:39:09.577 | DEBUG    | stubber.minify:minify_script:253 - Original length : 26905
2023-06-17 13:39:09.577 | INFO     | stubber.minify:minify_script:254 - Minified length : 11631
2023-06-17 13:39:09.577 | INFO     | stubber.minify:minify_script:255 - Reduced by      : 15274 
2023-06-17 13:39:10.854 | ERROR    | stubber.minify:cross_compile:330 - mpy-cross failed to compile:
MicroPy  Executing stubber on pyboard...
MicroPy  Pyboard  Writing /createstubs.mpy: |                                                                        | [0.00/0.00 @ ?B/s]
MicroPy  Pyboard  Traceback (most recent call last):
MicroPy  Pyboard  File "<stdin>", line 1, in <module>
MicroPy  Pyboard  ValueError: incompatible .mpy file
MicroPy  ✔ Done!
MicroPy  Copying stubs...
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/marcusstenbeck/.pyenv/versions/3.9.17/lib/python3.9/site-packages/micropy/app/stubs.py:18 │
│ 7 in stubs_create                                                                                │
│                                                                                                  │
│   184 │   log.success("Done!")                                                                   │
│   185 │   log.info("Copying stubs...")                                                           │
│   186 │   with tempfile.TemporaryDirectory() as tmpdir:                                          │
│ ❱ 187 │   │   pyb.copy_from(                                                                     │
│   188 │   │   │   DevicePath("/stubs"),                                                          │
│   189 │   │   │   tmpdir,                                                                        │
│   190 │   │   │   verify_integrity=True,                                                         │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │        _get_desc = <function stubs_create.<locals>._get_desc at 0x10ac0b9d0>                 │ │
│ │          backend = <CreateBackend.upydevice: 'upydevice'>                                    │ │
│ │          compile = True                                                                      │ │
│ │     create_stubs = <_io.BytesIO object at 0x10afeff40>                                       │ │
│ │              ctx = <click.core.Context object at 0x10ac98a00>                                │ │
│ │         dev_path = 'createstubs.mpy'                                                         │ │
│ │          exclude = []                                                                        │ │
│ │ exclude_defaults = True                                                                      │ │
│ │              log = <micropy.logger.ServiceLog object at 0x10ac6c9a0>                         │ │
│ │  message_handler = MessageHandlers(                                                          │ │
│ │                    │   on_message=<function stubs_create.<locals>.<lambda> at 0x10ac0baf0>,  │ │
│ │                    )                                                                         │ │
│ │           module = []                                                                        │ │
│ │  module_defaults = True                                                                      │ │
│ │               mp = <micropy.main.MicroPy object at 0x10ac6cb20>                              │ │
│ │             port = '/dev/cu.usbmodem1301'                                                    │ │
│ │              pyb = <micropy.pyd.pydevice.PyDevice object at 0x10ac984f0>                     │ │
│ │          pyb_log = <micropy.logger.ServiceLog object at 0x10ac98b80>                         │ │
│ │           tmpdir = '/var/folders/z0/s8bx3n216sl2s1qqg622rddc0000gn/T/tmpijehse71'            │ │
│ │          variant = <CreateStubsVariant.BASE: 'base'>                                         │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│                                                                                                  │
│ /Users/marcusstenbeck/.pyenv/versions/3.9.17/lib/python3.9/site-packages/micropy/pyd/pydevice.py │
│ :50 in copy_from                                                                                 │
│                                                                                                  │
│   47 │   │   src_path = Path(str(source_path))                                                   │
│   48 │   │   # 'is_dir/file' only works on existing paths.                                       │
│   49 │   │   if not src_path.suffix:                                                             │
│ ❱ 50 │   │   │   return self.pydevice.copy_dir(                                                  │
│   51 │   │   │   │   DevicePath(source_path),                                                    │
│   52 │   │   │   │   target_path,                                                                │
│   53 │   │   │   │   consumer=self.consumer,                                                     │
│                                                                                                  │
│ ╭────────────────────────────────────── locals ──────────────────────────────────────╮           │
│ │ exclude_integrity = {'usys.py', 'sys.py'}                                          │           │
│ │              self = <micropy.pyd.pydevice.PyDevice object at 0x10ac984f0>          │           │
│ │       source_path = '/stubs'                                                       │           │
│ │          src_path = PosixPath('/stubs')                                            │           │
│ │       target_path = '/var/folders/z0/s8bx3n216sl2s1qqg622rddc0000gn/T/tmpijehse71' │           │
│ │  verify_integrity = True                                                           │           │
│ ╰────────────────────────────────────────────────────────────────────────────────────╯           │
│                                                                                                  │
│ /Users/marcusstenbeck/.pyenv/versions/3.9.17/lib/python3.9/site-packages/micropy/pyd/backend_upy │
│ device.py:157 in copy_dir                                                                        │
│                                                                                                  │
│   154 │   │   target_path = Path(str(target_path))  # type: ignore                               │
│   155 │   │   source_path = self.resolve_path(source_path)                                       │
│   156 │   │   exclude_integrity = exclude_integrity or set()                                     │
│ ❱ 157 │   │   for file_path in self.iter_files(source_path):                                     │
│   158 │   │   │   rel_path = PurePosixPath(file_path).relative_to(                               │
│   159 │   │   │   │   list(PurePosixPath(file_path).parents)[-1]                                 │
│   160 │   │   │   )                                                                              │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │ exclude_integrity = {'usys.py', 'sys.py'}                                                    │ │
│ │            kwargs = {                                                                        │ │
│ │                     │   'consumer': <micropy.pyd.consumers.ConsumerDelegate object at        │ │
│ │                     0x10ac98ca0>,                                                            │ │
│ │                     │   'verify_integrity': True                                             │ │
│ │                     }                                                                        │ │
│ │              self = <micropy.pyd.backend_upydevice.UPyDeviceBackend object at 0x10ac98970>   │ │
│ │       source_path = '/stubs'                                                                 │ │
│ │       target_path = PosixPath('/var/folders/z0/s8bx3n216sl2s1qqg622rddc0000gn/T/tmpijehse71… │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│                                                                                                  │
│ /Users/marcusstenbeck/.pyenv/versions/3.9.17/lib/python3.9/site-packages/micropy/pyd/backend_upy │
│ device.py:140 in iter_files                                                                      │
│                                                                                                  │
│   137 │   │   if not results:                                                                    │
│   138 │   │   │   return                                                                         │
│   139 │   │   for file_result in results:                                                        │
│ ❱ 140 │   │   │   name, type_, _, _ = file_result                                                │
│   141 │   │   │   abs_path = PurePosixPath(path) / name                                          │
│   142 │   │   │   if type_ == stat.S_IFDIR:                                                      │
│   143 │   │   │   │   yield from self.iter_files(abs_path)                                       │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │ file_result = 'T'                                                                            │ │
│ │        path = '/stubs'                                                                       │ │
│ │     results = 'Traceback (most recent call last):\n  File "<stdin>", line 1, in              │ │
│ │               <module>\nOSError'+19                                                          │ │
│ │        self = <micropy.pyd.backend_upydevice.UPyDeviceBackend object at 0x10ac98970>         │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: not enough values to unpack (expected 4, got 1)

Looks like mpy cross compile is plain failing, and it's not possible to diable the cross compiling via command lien options either.

  • OS: macOS Ventura 13.2, M1 Pro processor
  • Micropy Version: 4.2.0 (g33b403dfb)
  • Python Version: Python 3.11.4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants