Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle failure from image converter service or other HttpErrors in wget() #128

Open
dhalbert opened this issue Sep 24, 2023 · 43 comments
Open

Comments

@dhalbert
Copy link
Contributor

dhalbert commented Sep 24, 2023

If wget() in PortalBase (https://github.com/adafruit/Adafruit_CircuitPython_PortalBase/blob/7a3277af2efb804c7ecf82d8bc87eb95c22dee2e/adafruit_portalbase/network.py#L319) gets some kind of HTTP error, it will raise adafruit_portalbase.network.HttpError, which is a subclass of Exception`.

This exception is not handled when wget() is called here:

self.wget(image_url, filename, chunk_size=chunk_size)
except OSError as error:
raise OSError(
"""\n\nNo writable filesystem found for saving datastream. Insert an SD card or set internal filesystem to be unsafe by setting 'disable_concurrent_write_protection' in the mount options in boot.py""" # pylint: disable=line-too-long
) from error
except RuntimeError as error:
.

A scenario where this happened is described in the forums here: the Cleveland Museum of Art may return image URL's that return 404's. These cause the image converter service to return a 422.

I'm thinking that maybe PortalBase should return an easier-to-catch exception, or PyPortal should catch that HttpError and turn it into something else.

@anglerfish27
Copy link

Hello brilliant minds!
I am just checking in to see if there are any updates on this. Doesn't look like it yet.
Any work around I could put in this file for the time being?

@anglerfish27
Copy link

Any luck. The CMA project code has gone from bad to worse. Now images don't load at all.
Having made no changes to the project. Still on 8.2.6 with 1.7.5 firmware for WiFi. Now I get the following when I plug in the Titano. I had to step away from this project for a bit, hence the delay as I was dealing with a family emergency.

Here's what I get now and I just the pre-stored CMA image that came with the project and it just sits there.

>>> %Run -c $EDITOR_CONTENT '`'
retrieving url: https://openaccess-api.clevelandart.org/api/artworks?cc0=1&has_image=1&indent=2&limit=1&type=Painting&skip=1381
Connecting to AP MartyMcW-Fi
Retrieving data...An error occured, retrying! - Sending request failed
loop counter: 0
Traceback (most recent call last):
  File "adafruit_requests.py", line 534, in _get_socket
  File "adafruit_requests.py", line 764, in connect
  File "adafruit_esp32spi/adafruit_esp32spi_socket.py", line 75, in connect
  File "adafruit_esp32spi/adafruit_esp32spi.py", line 806, in socket_connect
  File "adafruit_esp32spi/adafruit_esp32spi.py", line 702, in socket_open
  File "adafruit_esp32spi/adafruit_esp32spi.py", line 332, in _send_command_get_response
  File "adafruit_esp32spi/adafruit_esp32spi.py", line 288, in _wait_response_cmd
  File "adafruit_esp32spi/adafruit_esp32spi.py", line 197, in _wait_for_ready
TimeoutError: ESP32 not responding

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 51, in <module>
  File "/lib/adafruit_pyportal/__init__.py", line 305, in fetch
  File "adafruit_portalbase/network.py", line 518, in fetch
  File "adafruit_requests.py", line 823, in get
  File "adafruit_requests.py", line 668, in request
  File "adafruit_requests.py", line 515, in _get_socket
RuntimeError: Sending request failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 61, in <module>
KeyboardInterrupt: 
>>> 
Connection lost -- read failed: [Errno 6] Device not configured

Use Stop/Restart to reconnect.

Process ended with exit code 1.

@cogliano
Copy link
Contributor

cogliano commented Oct 17, 2023 via email

@anglerfish27
Copy link

Thank you @cogliano!

@cogliano
Copy link
Contributor

cogliano commented Oct 17, 2023 via email

@anglerfish27
Copy link

anglerfish27 commented Oct 23, 2023

Thank you @cogliano for taking a look. I know you were the creator of this project and I really appreciate your help. I gave it another try today and did a full nuke and pave.

I loaded CP 8.2.7 for the Titano. I nuked the SPI flash to ensure to no gremlins were left over. Firmware of the WIFI is updated as well to 1.7.5

I am still facing this "new" 2nd issue. The first issue was is that it would load the images but a random point it would encounter a URL that returned a 404 (you could manually click it to confirm, examples are in the Adafruit forums. While troubleshooting that I put this project aside due to some family issues. I hadn't touched a thing. All I did was turn it on. and now I'm presented with this "2nd" issue that essentially bricks the program.

I'm going to lay out every thing I did today so you can understand 100% my status.

Brand new Titano (well maybe 3 months old?). Like I said I nuked the flash and installed CP 8.2.7 (before it was 8.2.6 and before that I was given a link to 6.x to try!). I have the latest WiFi drivers for the ESP32 1.7.5.

Folder structure on the Titano:
/lib
/lib/adafruit_bitmap_font
/lib/adafruit_display_shapes
/lib/adafruit_io
/lib/adafruit_minimqtt
/lib/adafruit_pyportal
/lib/adafruit_touchscreen.mpy
/fonts/OpenSans-9.bdf

/background.bmp
/background_480.bmp
/boot_out.txt
/code.py
/secrets.py

I have tested this very set of files and successfully executed the NASA image of the day project, along with the Adafruit quote's project. So it is connecting just fine to Wifi and the SD card is working (I have tried so many SD cards of varying size and manufacturer's (FAT32 of course). I tried using both the different API URLs provided. No difference. My Adafruit API key is valid in my secrets file which is proved by the fact the other projects mentioned use it and work fine.

I just tried the code.py from github as well on the adafruit website. No difference.

code.py

import time
import random
import board
from adafruit_pyportal import PyPortal
from adafruit_display_shapes.circle import Circle

WIDTH = board.DISPLAY.width
HEIGHT = board.DISPLAY.height

#pylint: disable=line-too-long

# these lines show the entire collection
APIURL = "https://openaccess-api.clevelandart.org/api/artworks?cc0=1&has_image=1&indent=2&limit=1&skip="
IMAGECOUNT = 31954

# uncomment these lines to show just paintings
#APIURL = "https://openaccess-api.clevelandart.org/api/artworks?cc0=1&has_image=1&indent=2&limit=1&type=Painting&skip="
# IMAGECOUNT = 3223

BACKGROUND_FILE = "/background.bmp"
if WIDTH > 320:
    BACKGROUND_FILE = "/background_480.bmp"

pyportal = PyPortal(default_bg=BACKGROUND_FILE,
                    image_json_path=["data", 0, "images", "web", "url"],
                    image_dim_json_path=(["data", 0, "images", "web", "width"],
                                         ["data", 0, "images", "web", "height"]),
                    image_resize=(WIDTH, HEIGHT - 15),
                    image_position=(0, 0),
                    text_font="/fonts/OpenSans-9.bdf",
                    json_path=["data", 0, "title"],
                    text_position=(4, HEIGHT - 9),
                    text_color=0xFFFFFF)

circle = Circle(WIDTH - 8, HEIGHT - 7, 5, fill=0)
pyportal.splash.append(circle)
loopcount = 0
errorcount = 0
while True:
    response = None
    try:
        circle.fill = 0xFF0000
        itemid = random.randint(1, IMAGECOUNT)
        # itemid = 20 # portrait mode example
        # itemid = 21 # landscape mode example
        print("retrieving url:", APIURL + str(itemid))
        response = pyportal.fetch(APIURL + str(itemid))
        circle.fill = 0
        print("Response is", response)
        loopcount = loopcount + 1

    except (RuntimeError, KeyError, TypeError) as e:
        print("An error occured, retrying! -", e)
        print("loop counter:", loopcount)
        assert errorcount < 20, "Too many errors, stopping"
        errorcount = errorcount + 1
        time.sleep(60)
        continue

    errorcount = 0
    stamp = time.monotonic()
    # wait 5 minutes before getting again
    while (time.monotonic() - stamp) < (5*60):
        # or, if they touch the screen, fetch immediately!
        if pyportal.touchscreen.touch_point:
            break

Upon saving and applying power to the Titano I get the CMA 480.bmp image displayed. That's where it sits there forever never progressing. Thonnny shows...:

retrieving url: https://openaccess-api.clevelandart.org/api/artworks?cc0=1&has_image=1&indent=2&limit=1&type=Painting&skip=18017 Connecting to AP MartyMcW-Fi Retrieving data...An error occured, retrying! - Sending request failed loop counter: 0``

It will continue this error count until I manually control+C to break it, when I do Thonny throws the following:

`>>> %Run -c $EDITOR_CONTENT '`'
retrieving url: https://openaccess-api.clevelandart.org/api/artworks?cc0=1&has_image=1&indent=2&limit=1&type=Painting&skip=18017
Connecting to AP MartyMcW-Fi
Retrieving data...An error occured, retrying! - Sending request failed
loop counter: 0

Traceback (most recent call last):
  File "adafruit_requests.py", line 534, in _get_socket
  File "adafruit_requests.py", line 764, in connect
  File "adafruit_esp32spi/adafruit_esp32spi_socket.py", line 75, in connect
  File "adafruit_esp32spi/adafruit_esp32spi.py", line 806, in socket_connect
  File "adafruit_esp32spi/adafruit_esp32spi.py", line 702, in socket_open
  File "adafruit_esp32spi/adafruit_esp32spi.py", line 332, in _send_command_get_response
  File "adafruit_esp32spi/adafruit_esp32spi.py", line 288, in _wait_response_cmd
  File "adafruit_esp32spi/adafruit_esp32spi.py", line 197, in _wait_for_ready
TimeoutError: ESP32 not responding

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 51, in <module>
  File "adafruit_pyportal/__init__.py", line 306, in fetch
  File "adafruit_portalbase/network.py", line 518, in fetch
  File "adafruit_requests.py", line 823, in get
  File "adafruit_requests.py", line 668, in request
  File "adafruit_requests.py", line 515, in _get_socket
RuntimeError: Sending request failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 61, in <module>
KeyboardInterrupt: `

I'm willing to do whatever is needed. I was really happy when it was working (so I thought) until it started hitting 404 URLs when it hit one it would freeze on whatever the last image displayed was. I bought the Titano for this project. I'm trying here I really am. I don't know what I'm doing wrong. Maybe your code is different than mine? I dunnno. $50+shipping and I can't even run the project :(

I have the smaller version of the Titano the PyPortal (regular) which is running the OpenWeather API project on Adafruit's site. That one works awesome too. It does crash every few days on socket issues. I'm going to try upgrading the code on it and see if I can debug it myself too. I don't mind having to push the reset button a few times every few days.

When you said it "works for you" how much time did you let it run? To call it stable you would probably need to let it run at least 3-4 days 24/7. Or maybe you fell into the trap I did and thought it was working great until I noticed images were not changing after awhile leading to this ticket and the whole 404 image issue.

Please help I really want to get this project working and stable. I have a nice picture frame I cut out a nice matted layer square to place it in and hide everything behind it so I can hang it on a wall. Kinda need the thing to work though :)

Thank you for your help and I am standing by on next steps..
Respectfully,
Anglerfish27

@anglerfish27
Copy link

If I wait long enough the LCD will display the following: (shortened but keeping the key points)

esp32spi.py line 197 in _wait_for_ready

The above exceptio was the direct cause of the following exception:
code.py line 51
/pyportal/__init__.py line 306 in fetch
/portalbase/network.py 518 inn fetch
adafruit_requests.py line 823 in get
adafruit_requests.py line 668 in request
adafruit_requests.py line 515 in _get_socket
Runtime error sending request failed.
Trace back last call line in code.py line 59
AssertationError Too many errors stopping.

@FoamyGuy
Copy link
Contributor

@anglerfish27 I have been doing lots of testing around this issue and the cleveland art pyportal project to try to understand the issues you're facing.

It looks like you've faced a few different symptoms and I don't have answers or information about all of them, but I can try to help get you back to a state where the project can work.

In your second most recent message you had this error message:

Traceback (most recent call last):
  File "adafruit_requests.py", line 534, in _get_socket
  File "adafruit_requests.py", line 764, in connect
  File "adafruit_esp32spi/adafruit_esp32spi_socket.py", line 75, in connect
  File "adafruit_esp32spi/adafruit_esp32spi.py", line 806, in socket_connect
  File "adafruit_esp32spi/adafruit_esp32spi.py", line 702, in socket_open
  File "adafruit_esp32spi/adafruit_esp32spi.py", line 332, in _send_command_get_response
  File "adafruit_esp32spi/adafruit_esp32spi.py", line 288, in _wait_response_cmd
  File "adafruit_esp32spi/adafruit_esp32spi.py", line 197, in _wait_for_ready
TimeoutError: ESP32 not responding

I believe the root cause of this error is: adafruit/nina-fw#58

Unfortunately there is some underlying issue within nina-firmware (which is used by the ESP32 on the pyportal) that results in this error occuring when attempting to communicate with a server that uses a certain type of HTTPS certificate. I confirmed with Dan (the one who dug into this when it was noticed) that the cleveland art API is using a certificate of the type that there are problems with.

However I know that this project was working at one point in time, you had it running on your device for a while until some unrelated 404 issues popped up. I have run it on my devices at various times as well.

It turns out that the nina firmware that was pre-loaded on the ESP32 that's on the pyportal does not seem to have the same problem. So that "out of the box" firmware can successfully make requests and get responses from the cleveland art API. It proved to be a bit of a challenge to track down a copy of the bin file for that original firmware, but Dan helped with that and was able to pull a copy from a device he has.

I've loaded this bin on my pyportal Titano and tested it successfully with the cleveland art project. My device has circuitpython 8.2.7 loaded on it and the ESP32 has this specific firmware loaded and it's allowing the project to work.

Link to .bin file:
https://cdn.discordapp.com/attachments/1166371166806622210/1166531169383350344/nina-fw-122-from-titano.bin?ex=654ad3cf&is=65385ecf&hm=028f7970b16d3ca3a40754afc40cf3a0ddb3551ebc9b1f2d9848a004284e1066&

It looks like you're already familiar with the process of upgrading firmwares on the ESP32 because you mentioned loading the latest one. The process to load this one should be the exact same, just choose this bin file instead of any others.

Ordinarily upgrading the firmware would be the thing you want to do, but in this case unfortunately there was an issue that came up at some point with this type of certificate and there isn't currently a fix for it in any newer versions of the firmware. So right now the best work around to get the art project running is to use this version of the esp32 firmware for the co-processor.

I read through the thread here, on the adafruit forums, and in the issue you made on the cleveland art repo. It sounds like the initially reported 404 issues may have been alleviated on their side somehow. But I do also think it's possible for us to have better error handling for that in the project code that can have it choose a new number and try loading a new artwork. I'm going to leave my device running the art project for a while to see if I encounter any further issues with it. If I do I'll report back with information about them and hopefully potential fixes.

Please let me know if you have further issues or need help with the process to get this firmware version loaded. I am willing to try to help you get your pyportal back to running the art project, feel free to ping me here.

@anglerfish27
Copy link

Wow thank you @FoamyGuy. I'm speechless honestly. This level of effort from everyone involved is really giving me hope for humanity!! :) Thank you for the deep dive explanation. I am going to give the older WiFi a try and hopefully it works. Any hopes or ETA on a 1.7.6 FW from ExpressIF that will have ''all the things"? lol I know you don't control that.

Yeah the 404's where interesting until I tracked down what was happening at least within what I could see (I'm not super MP smart, just enough to poke at MP' libraries if needed). I opened a ticket with the CMA GitHub team as well around the 404's which they say should be fixed. I wonder if its of any value mentioning the issue you found with their cert type to them there? I mean the API is mostly going to be used by WiFi enabled devices, I doubt a lot of people use it on their computer so I would opine there's a good chance an ESP32 is involved and hindering their project from being more friendly to devices. Granted there are many other wifi modules. I dunno I'm thinking like my sysadmin days. Cert replacing can take 5 minutes or implode your entire environment if you don't know what you are doing!

Granted if CMA is doing something very odd with certs (very curious what you found in the certs as strange) maybe they would be thankful to know about it. Just a thought. The Github to CMA is https://github.com/ClevelandMuseumArt/openaccess/issues/8#issuecomment-1766889912.

Alright lets hope I don't brick this thing with the firmware downgrade! Going to get started on it now. (fingers crossed)

@dhalbert
Copy link
Contributor Author

There isn't anything wrong with CMA's cert chain: they are using a Let's Encrypt cert chain with an ECDSA cert (elliptical). For some reason, NINA-FW has stopped handling these properly, sometime after 1.2.2. These are going to become more and more common, so we would like to make them work.

The proper compile options appear to be turned on in 1.7.x, so we are unsure what is happening. It is not a question of a missing root cert. We are going to compare the sources for 1.2.2 and 1.7.x and see if we can discern anything.

@anglerfish27
Copy link

Well good news team! CMA project is up and running on the NINA-FW 1.2.2 with CP 8.2.6!

It's been running for awhile now shortly after my post as it didn't take long to flash everything.
Thanks for the details about the cert, I figured there was nothing wrong with their cert since its valid, but I was curious what part of it (now I know the elliptical) part is what breaks the NINA-FW.

I'm going to let it continue to bake for the next few days and see how it holds up or if it jams up or not.

I may be opening a separate case for the regular Pyportal and the open weather project. I keep getting socket failures every few days. I know that has 1.7.5 for NINA-FW as well. But its a few revs back on CP. Maybe that will help. Anyways that's not for this case ticket/forum.

SO FAR SO GOOD!! I think if by tomorrow its still working we have fully root caused it.
-Hoping the enhanced error handling of the 404's is added to future code along with a new firmware that deals with these certs properly but I know that isn't on you guys so much.

Recommend updating the Titano documentation, at least for CMA and warn NOT to upgrade the NINA-FW.

Thank you so much. If you need anything from me let me know as well.
Cheers,
Anglerfish

@anglerfish27
Copy link

Well it was going so well. No crash all day long ...until..now. New error. Unfortunately It was just connected to USB power not my computer so I don't have any output from thonny. What you see is all I have. I checked the SD card immediately. It was clean, empty except for the image cache file it creates. Based in the msg maybe the image itself had something to do with it processing incorrectly? I dont know. I guess I will connect Thonny to it and let it run over night so if an error is thrown I might get more info. See attached message.

crash

@anglerfish27
Copy link

I just noticed the URL is in there at the top!. Might help for a repro..

@anglerfish27
Copy link

https://openaccess-cdn.clevelandart.org/2003.105/2003.105_web.jpg

Image loads fine on my browser.

@anglerfish27
Copy link

looking at how the code pushes the image URL to send it to Adafruit for reprocessing I tried running that URL and it worked, sort of. Meaning it gave me a .BMP to download. I was unable to open the image (which matched the size of the SD card cache size) with "Preview" in my Mac. However I was able to open the image in GIMP as well as Sublime text (I wanted to see if there would be text and instead displayed the image!). Other imaging program can also open it...

https://io.adafruit.com/api/v2/anglerfish27/integrations/image-formatter?x-aio-key=**MY-SECRET-ADAFRUIT-KEY**&width=193&height=305&output=BMP16&url=https://openaccess-cdn.clevelandart.org/2003.105/2003.105_web.jpg

I guess if someone can get into the guts of the program and feed it this we should see if it errors. I can't attach the BMP file that it gave me to download. Github wont let me, and if I change the extension then you just get the image displayed. try it with the above URL and your IO passcode. Something about that image made the program angry. I have thonny running it now overnight to see what happens. If I get it again. I will switch out the SD card though I doubt that's the problem. I dont know enough where in the code flow to "inject" this URL as to replicate the issue.

@anglerfish27
Copy link

code snips start from the bottom up of the error lines...(so first one is _blit)

code around 474

        def _blit(
        self,
        bitmap: displayio.Bitmap,  # target bitmap
        x: int,  # target x upper left corner
        y: int,  # target y upper left corner
        source_bitmap: displayio.Bitmap,  # source bitmap
        x_1: int = 0,  # source x start
        y_1: int = 0,  # source y start
        x_2: int = None,  # source x end
        y_2: int = None,  # source y end
        skip_index: int = None,  # palette index that will not be copied
        # (for example: the background color of a glyph)
    ) -> None:
        # pylint: disable=no-self-use, too-many-arguments

        if hasattr(bitmap, "blit"):  # if bitmap has a built-in blit function, call it.   **####THIS IS LINE 474####**
            # this function should perform its own input checks
            bitmap.blit(
                x,
                y,
                source_bitmap,
                x1=x_1,
                y1=y_1,
                x2=x_2,
                y2=y_2,
                skip_index=skip_index,
            )

line 448

                        self._blit(
                        bitmap,
                        xposition + my_glyph.dx,
                        y_blit_target,
                        my_glyph.bitmap,
                        x_1=glyph_offset_x,
                        y_1=y_clip,     **#####THIS IS LINE 448####**
                        x_2=glyph_offset_x + my_glyph.width,
                        y_2=my_glyph.height,
                        skip_index=skip_index,  # do not copy over any 0 background pixels
                    )

line 209

            # Place the text into the Bitmap
            self._place_text(
                self._bitmap,
                text if self._label_direction != "RTL" else "".join(reversed(text)),
                self._font,
                self._padding_left - x_offset,
                self._padding_top + y_offset, **#######LINE 209########**
            )

            if self._base_alignment:
                label_position_yoffset = 0
            else:
                label_position_yoffset = self._ascent // 2

line 544 (final one from bitmap label)


    def _set_line_spacing(self, new_line_spacing: float) -> None:
        if self._save_text:
            self._reset_text(line_spacing=new_line_spacing, scale=self.scale) **###LINE 544###**
        else:
            raise RuntimeError("line_spacing is immutable when save_text is False")

Moving to init.py line 420

    @text.setter  # Cannot set color or background color with text setter, use separate setter
    def text(self, new_text: str) -> None:
        self._set_text(new_text, self.scale). **####LINE 420####**

Moving to 298 in PORTALBASE/init.py


                if index_in_splash is not None:
                    self.splash[index_in_splash] = self._text[index]["label"]
                else:
                    self.splash.append(self._text[index]["label"])
            else:
                self._text[index]["label"].text = string.        **###LINE 298####**
            self._text[index]["label"].color = self._text[index]["color"]
            self._text[index]["label"].anchor_point = self._text[index]["anchor_point"]

next up 432 in portalbase init

       # For backwards compatibility
        if isinstance(alarms, (float, int)):
            alarms = self.create_time_alarm(alarms)

        self._alarm.light_sleep_until_alarms(alarms)

    def _fetch_set_text(self, val, index=0):
        self.set_text(val, index=index). **###LINE 432###**

line 484 same file..

    def _fill_text_labels(self, values):
        # fill out all the text blocks
        if self._text:
            value_index = 0  # In case values and text is not the same
            for i in range(len(self._text)):  # pylint: disable=consider-using-enumerate
                if (not self._text[i]["is_data"]) or (value_index > (len(values) - 1)):
                    continue
                string = None
                if self._text[i]["transform"]:
                    func = self._text[i]["transform"]
                    string = func(values[value_index])
                else:
                    try:
                        string = "{:,d}".format(int(values[value_index]))
                    except (TypeError, ValueError):
                        string = values[value_index]  # ok it's a string
                self._fetch_set_text(string, index=i). **###LINE 484####**
                value_index += 1

lastly pyportal/init line 357

        # if we have a callback registered, call it now
        if self._success_callback:
            self._success_callback(values)

        self._fill_text_labels(values). **###LINE 357###**
        # Clean up
        json_out = None
        response = None
        gc.collect()

all the way to the inception in code.py

response = pyportal.fetch(APIURL + str(itemid))

I hope these little code snips and the processing url will help someone narrow down the issue faster. That was my intent with sharing the snips. You all are way smarter than me and can probably decipher the puzzle quickly!

Much respect,
Anglerfish27

@FoamyGuy
Copy link
Contributor

@anglerfish27 Thanks for posting those ships chasing this issue down.

The root cause of this is unrelated to our prior problems. It's a bug in the BitmapLabel class interacting with a quirk of certain characters of the open-sans-9 font used by this project.

I've submited a PR over in the display_text library with a fix for this issue: adafruit/Adafruit_CircuitPython_Display_Text#194

If you want to give this a try on your device you can either make the same change in your local copy of the library, or download the one from the PR branch and put it on your device.

Note that adafruit_display_text library is frozen in to the pyportal build, which means if you want to use a customized version (which we need to do at the moment for this fix) then you have to put the library in the root of CIRCUITPY rather than the lib folder

So you should have:

CIRCUITPY/
    lib/
        <libraries>...
    boot_out.txt
    adafruit_display_text/
    code.py

not the more typical:

CIRCUITPY/
    lib/
        adafruit_display_text/
        <libraries>...
    boot_out.txt
    code.py

@anglerfish27
Copy link

anglerfish27 commented Oct 26, 2023

Thanks @FoamyGuy you are awesome man! It did freeze overnight. However I guess it has not reached a point where it displays the error (on the LCD or Thonny). I am going to make the code change and let it run again. I feel bad bugging you guys (and gals) by running into all these minor (but show stoppers) bugs but I guess its good they are getting fixed for the CircuitPython language as a whole!

Much respect - will report back tomorrow if it its frozen or not. Since its a one liner, I'm going to just make the change myself and put the .py in the / partition as you explained above. Fingers crossed again!! :)

Cheers,
Anglerfish27

@anglerfish27
Copy link

Alright I have it hooked up to one of my home made battery back that should allow it to run for well over a week! With a little luck I'll report back that its running fine tomorrow and also that my battery pack didn't explode into flames! I'm taking safety precautions I promise!!

@anglerfish27
Copy link

anglerfish27 commented Oct 27, 2023

Bad news bears @FoamyGuy :(
It crashed again, the error was the same but the values were different this time it was
ValueError: x must be in 0-1154

See attached image. I am running the modified code, likely explains why the value changed from 300 to 1154.
Unfortunately the way the program spit out on the LCD I cannot see the URL(image) that triggered it.

Standby by to try anything out as always!
crash2

@anglerfish27
Copy link

Any thoughts folks, it runs logger now for sure but it still crashes every day the X range changes too its not always the same sometimes its lower ect..

@FoamyGuy
Copy link
Contributor

@anglerfish27 this is manifestation of a similar bug to the one that was fixed within bitmap label. It appears that there are more cases within the font that can trigger similar situations of trying to put pixels "out of bounds".

Basically certain characters from this font drawn at certain positions within the string (likely first and last position, but possible others) are triggering this bug. I'll need to track down the trigger this time in order to fix it, my best guess is that it's coming from the last character in the string, but I don't know for certain without more testing.

I will try to figure out the cause of the remaining issues and work on a patch over in bitmap_label for them.

If you happen to have had serial console open and have access to the JSON object which was pulled just before the error occurred it can help me to more quickly narrow down what is triggering the bug now. No worries if not. I'll try to come up with a way to cycle through the possible objects from the API to try to identify the problematic ones.

In the meantime something you can try if you'd like is changing the font used by the project to be the default system font instead of the Opensans-9 bdf file. I think you can comment out the line here: https://github.com/adafruit/Adafruit_Learning_System_Guides/blob/main/PyPortal_CMA_Art_Frame/code.py#L34 to use the default font. To the best of my knowledge the default font does not have the same issues.

@anglerfish27
Copy link

Shoot, I've been running the project off battery power so it was not connected to Thonny, today and yesterday seem especially bad. I'm going to hook it up to Thonny and let it run to get you the info that will help. I'll hold off on changing the font to allow us to get some diagnostic data to help work on this. if there is something I can do to the device to get more logs somehow, let me know I'll be glad to do so. I just kicked it off to run w/Thonny. When I get an error I'll share it asap.

Thanks!
Anglerfish27

@FoamyGuy
Copy link
Contributor

@anglerfish27 Thank you! I've got mine running as well in the hopes of coming across one of the ones that triggers the out of bounds bug. It'll be down to luck a little bit with the way it's coded since they're chosen at random. Maybe I'll try to tweak to go sequentially instead to make the testing a little more ordered and remove chances for dupes.

The main two things I can think of that will help:

  • make sure your version of the project code still has this print statement for the URL: https://github.com/FoamyGuy/Adafruit_Learning_System_Guides/blob/main/PyPortal_CMA_Art_Frame/code.py#L50 that should be printing the URLs to the cleveland API as the requests are about to get made. So when it finds one with the problematic title and crashes, the most recently printed API URL should be the one that triggered it. From the API URL I'll be able to get the title string and hopefully make a reproducer script for the bug that doesn't rely on the API at all, then fix the root cause.
  • Is your version of the project configured to look at only Paintings? or do you have the default configuration of all art types? That is determined by the variables here: https://github.com/FoamyGuy/Adafruit_Learning_System_Guides/blob/main/PyPortal_CMA_Art_Frame/code.py#L16-L22 There number of artworks for specifically Paintings is much lower, so if you were using that type when you had it fail in the past I'll switch over to use Paintings also. With the lower number of them presumably it should "easier" to find one with the current random selection.

@anglerfish27
Copy link

anglerfish27 commented Oct 30, 2023

Hey @FoamyGuy , yes the line 50 is in place so I get all the URLs dumped to Thonny, for example:

retrieving url: https://openaccess-api.clevelandart.org/api/artworks?cc0=1&has_image=1&indent=2&limit=1&type=Painting&skip=2052
Retrieving data...Reply is OK!
image dim: 900 134
original URL: https://openaccess-cdn.clevelandart.org/1938.301.89.a/1938.301.89.a_web.jpg
convert URL: https://io.adafruit.com/api/v2/anglerfish27/integrations/image-formatter?x-aio-key=aio_UsZd79z4kAGDl1Mk8CpF10Zr8Amw&width=480&height=305&output=BMP16&url=https://openaccess-cdn.clevelandart.org/1938.301.89.a/1938.301.89.a_web.jpg
Fetching stream from https://io.adafruit.com/api/v2/anglerfish27/integrations/image-formatter?x-aio-key=xxxxxxxw&width=480&height=305&output=BMP16&url=https://openaccess-cdn.clevelandart.org/1938.301.89.a/1938.301.89.a_web.jpg
Reply is OK!
Saving data to  /sd/cache.bmp

This is not one of the failure URL conditions, but representative of what we can expect to get on a failed case.
I am indeed using the paintings only URL that was commented out in the vanilla code base
I'm using:

APIURL = "https://openaccess-api.clevelandart.org/api/artworks?cc0=1&has_image=1&indent=2&limit=1&type=Painting&skip="

IMAGECOUNT = 3223

When I am not tied up I also touch the screen to force a cycle to a new image to see if we can hit the issue faster. Figures its been running smooth after 2 failures this morning..sigh. We'll get it!

@anglerfish27
Copy link

Darn it I always forget about the secret key apologies. Thanks for killing it. I have generated a new one and the project is running in test mode to get the data we need. 👍

@anglerfish27
Copy link

anglerfish27 commented Oct 31, 2023

Checked on it this morning, I'm not sure if the crash that it hit right now is because my laptop had its lid closed and I opened it up and logged in (OS X) thonny was still running I could see the previous images it loaded....but I dont know if that transition of the laptop going from an idle state to active state with Thonny cause it or not. This is not related to the font issue clearly.

............................................................................................................................................................................................................................................................................................................Created file of 292938 bytes in 21.1 seconds
Response is The Nodding Stone Terrace, Tiger Hill, and the Thousand-Man Seat, from Twelve Views of Tiger Hill, Suzhou
retrieving url: https://openaccess-api.clevelandart.org/api/artworks?cc0=1&has_image=1&indent=2&limit=1&type=Painting&skip=226
Retrieving data...An error occured, retrying! - Sending request failed
loop counter: 6
retrieving url: https://openaccess-api.clevelandart.org/api/artworks?cc0=1&has_image=1&indent=2&limit=1&type=Painting&skip=3142
Connecting to AP MartyMcW-Fi
Retrieving data......................Created file of 292938 bytes in 21.1 seconds
Response is The Nodding Stone Terrace, Tiger Hill, and the Thousand-Man Seat, from Twelve Views of Tiger Hill, Suzhou
retrieving url: https://openaccess-api.clevelandart.org/api/artworks?cc0=1&has_image=1&indent=2&limit=1&type=Painting&skip=226
Retrieving data...An error occured, retrying! - Sending request failed
loop counter: 6
retrieving url: https://openaccess-api.clevelandart.org/api/artworks?cc0=1&has_image=1&indent=2&limit=1&type=Painting&skip=3142
Connecting to AP MartyMcW-Fi
Retrieving data...

Will see if I catch the font error again today. Seems like being hooked up to Thonny changes the frequency. Felt like yesterday it was happened often on battery. But depending on how the SD card cache proces goes (.......) it may cut off some or all of the much needed URL on the LCD if we were to try battery power.

Right now Thonny isn't actually crashed and I have the red dot like its trying to get data on the LCD. Touching doesn't help either. Going to "pull the cord" and start over. Sigh. That URL is fine too.

Maybe I have to set my laptop to never "sleep" over night. Or run on battery power and "hope" we get one where the URL is still legible. OH now that I think about it I may have some in my deleted photos. Will post if they have URLS! No promises.

@anglerfish27
Copy link

Darn, nope. I have one thats similar but due to the SD card cache spewing so many ......... the URL is lost :( nothing of value to help us on it.

@anglerfish27
Copy link

anglerfish27 commented Nov 1, 2023

@FoamyGuy I got one finally. Thonny wasn't running but fortunately the LCD has the URL. See image.
IMG_0594

https://openaccess-cdn.clevelandart.org/2003.135.b/2003.135.b_web.jpg

image loads for me in a web browser.

@anglerfish27
Copy link

Got hit with another one over night. Hope this helps! @FoamyGuy
IMG_0599

@anglerfish27
Copy link

anglerfish27 commented Nov 2, 2023

Here's the URL so don't have to strain your eyes to read it. It loads. The image is uh ahem adults only. But its art so.. :)

https://openaccess-cdn.clevelandart.org/1966.387/1966.387_web.jpg

@FoamyGuy
Copy link
Contributor

FoamyGuy commented Nov 5, 2023

@anglerfish27 thank you for sharing the link! I was able to track down that artwork in the API and modify my version of the CMA script to load it specifically. This is the API URL that fetches it: https://openaccess-api.clevelandart.org/api/artworks/?accession_number=1966.387

The title of this artwork is "July" which starts with the capital J and appears that your error is the same one as before which should now be fixed in the display_text library.

I verified on my device using the currently released latest version of adafruit_display_text my device is able to show the artwork and title without that error being raised.

Try deleting the adafruit_display_text/ directory on your device and then downloading the newest one from the current bundle and replacing the one you had. Perhaps at some point the "wires got crossed" and you ended up without the one that has the max() fix for boundaries in it.

adafruit_display_text is one of the frozen in libraries as well so you'll need to place the currently released version of the library in the root of your device instead of inside lib/ like is more typical for libraries.

It should be located at CIRCUITPY/adafruit_display_text/

If you want to remove the CMA project from the equation for the moment you can use this reproducer script to directly create one of the labels that were problematic:

import board
from adafruit_display_text.bitmap_label import Label
from adafruit_bitmap_font import bitmap_font

display = board.DISPLAY
font_file = "fonts/OpenSans-9.bdf"

# Set text, font, and color
text = "July JJJ"
font = bitmap_font.load_font(font_file)
color = 0xFF00FF

# Create the tet label
text_area = Label(font, text=text, color=color)
text_area.x = 20
text_area.y = 20
display.root_group = text_area

while True:
    pass

The older version of display text without the fix will crash with this script, the newer is able to display it to successfully.

@anglerfish27
Copy link

Awesome thank you @FoamyGuy . I didn't forget about you or this project. I just had some family/life issues I'm dealing with that have taken my time to work on this. I am going to take the suggested action plan and see how it goes. I'm going to download the latest files and give it a go. I love that you narrowed down the issue to a script we can use to verify! Sweet! Going to do some testing..more to come..

Happy thanksgiving for those in the U.S. :)
Cheers,
Anglerfish27

@anglerfish27
Copy link

anglerfish27 commented Nov 24, 2023

Man that script is super handy. I ran it against my current code and yep sure it enough it bombed out as expected. I then read your message too quick about displaytext and put all of those files in the root of the drive. That didn't work. That's when I re-read your message more closely. I added them as /adafruit... and the test failed but differently. It said it could not find adafruit_display_text.bitmap_label seemed to me like a directory tree issue.

I decided to throw bitmap_label.py up into the root of the drive (leaving a copy in the display_text) folder as well. I re-ran the test, boom no errors!

So now I'm stress testing it against the real deal. So far its going smooth as silk. I am going to let it marinade at least until tomorrow. Hopefully no more surprises! Thank you again. I will update you good bad or otherwise!

Cheers mate,
Anglerfish27

@anglerfish27
Copy link

Well after letting it run for a few days connected just to USB it appeared to be working. I noticed one odd behavior, it was the initial CMA background image, after a few seconds it loaded a painting. I dont know if it rebooted itself or what. I left it running (it was still working even with this CMA background quirk) and then poof. It froze. I could not get it to load a new image by touching the screen. It was just frozen on an image. Since there was no IDE connected (just plain USB pwr) no error messages for me to share. So I hooked it up to a junker computer with Thonny on it and let it run via code.py and let it do its thing. Worked fine for what was less than 24 hours and then same issue. Its stuck on an image (it has the little red dot on the bottom indicating its processing/downloading). But this time I have the Thonny output in the REPL. I will share that in a bit as its on another computer and I need to step away for an hour. The very last error was HttpError: Code: 422 Unprocessable Entity. . I will post the full message when I can but it looks like line 292 in wget 191 process image and 336 in fetch. I have the image url as well.

@anglerfish27
Copy link

@FoamyGuy

`retrieving url: https://openaccess-api.clevelandart.org/api/artworks?cc0=1&has_image=1&indent=2&limit=1&type=Painting&skip=217

Retrieving data...Reply is OK!

image dim: 1241 893

original URL: https://openaccess-cdn.clevelandart.org/1944.79/1944.79_web.jpg

convert URL: https://io.adafruit.com/api/v2/anglerfish27/integrations/image-formatter?x-aio-key=xxxxx&width=480&height=305&output=BMP16&url=https://openaccess-cdn.clevelandart.org/1944.79/1944.79_web.jpg

Fetching stream from https://io.adafruit.com/api/v2/anglerfish27/integrations/image-formatter?x-aio-key=xxxxx&width=480&height=305&output=BMP16&url=https://openaccess-cdn.clevelandart.org/1944.79/1944.79_web.jpg

Traceback (most recent call last):
File "", line 50, in
File "adafruit_pyportal/init.py", line 336, in fetch
File "adafruit_pyportal/network.py", line 191, in process_image
File "adafruit_portalbase/network.py", line 292, in wget
HttpError: Code 422: Unprocessable Entity`

@makermelissa
Copy link
Contributor

This is an HTTP error returned by the server, rather than an error with the code. Most likely the original url returned something that wasn't an image to the image formatter, which caused it to throw that error. These server returned http errors cause an HttpError exception to be thrown by the library so that your code can catch it and respond appropriately.

@anglerfish27
Copy link

So is this the original issue? This isn't "my" code it came from Adafruit's website as a project. I purchased all the hardware from them to make this project and its been a battle ever since.

I'm glad to make code changes but it sounds like those changes need to happen in the core libraries from Adafruit, that being said I'm stilling willing to make the changes in the .py version of the file and run them but someone way smarter than me has to tell where in the code this isn't being handled right and how what to add.

That's beyond my python knowledge. Feel awfully similar to the original issue. @FoamyGuy and others have been amazing helping get the project the stable because along the way we have found ahem other issues.

I've still got this $60 Titano in my homelab. It was for awhile going to be gift, that time past, now I want to frame it with a nice matte black around the edge to hide most of the LCD and frame with a battery. I haven't bothered to even order a frame because I feel we are never going to get to the bottom of this.

$60 and months now of hard work (no bashing the team helping you are all AMAZING!) I just wish I could have it working already :(. Any and all help from @FoamyGuy and others would be super awesome. Going through a rough time in my life with my family, details are not necessary, and I guess I am projecting some of that worry/stress/anxiety onto this project.

I know the team is busy coding and fixing more important issues, but I do feel if Adafruit is going to sell a device and market on their website a complete project with code and instructions, that they need to stand behind it until its fully fixed or well a refund I guess.. I continue to hold out hope the amazing team here working this issue.

My full respect,
Anglerfish27

@makermelissa
Copy link
Contributor

Ah yes, I just reread the original issue. @dhalbert this was intentional and code like the Cleveland Art example should place this in a try/except block. That's the reason it's a different error type so that the top level code can handle.

For the Cleveland Art Museum project, the correct fix would be to add HttpError to https://github.com/adafruit/Adafruit_Learning_System_Guides/blob/main/PyPortal_CMA_Art_Frame/code.py#L56

@anglerfish27
Copy link

Thanks @makermelissa yeah this seems like the original issue. It's happen quite often now. I'm on CP 8.2.7.

I see CP is now up to 8.2.9. Is there any value add in terms of solving the stability issue of this project for me to upgrade to CP 8.2.9? Or is it just still going to be there.

You mention the code needs a try/except block. Can you tell me which code needs to be in this block? is the adafruit_portalbase/network.py somewhere around line 292 in wget? I'm learning python for awhile so I can do minor things to existing code. I don't know if I can pull this off. I dont know what it's supposed to do if the exception is triggered, meaning I don't know where it would need to go in the code from there. Reboot the darn thing? I know thats a hammer approach.

Which is currently what I did for the openweather project (on a regular Pyportal)Adafruit also sells/documents. I forget where in the code I told it to reboot if the error message was found due to failure of retrying sockets if I recall. It mostly works(stable for a few days usually), though its acting weird still. For example if I simple push the reset button on it, the screen goes black and stays that way. I have physically unplug power to it and reconnect it. My guess is the wireless is left is some screwed up state it just jams up. I haven't devoted time to that other than the reboot. I may try and throw that in this code and see what happens. When its working its awesome (other than the PRNG in CP is terrible, I see a lot of the same images over and over, I'm going to research how I can make the PRNG more "random" maybe with a salt I dunno.

Anyways back to the issue. What's the plan? How can we fix it? or work around it. I don't mind running ghetto code that is ugly if it works. I really want this project to work. Some of the paintings are downright beautiful and the higher res of the Titano makes them look awesome!. As always I'm ready and willing to try/test whatever the team wants.

Respectfully,
Anglerfish27

@anglerfish27
Copy link

anglerfish27 commented Dec 15, 2023

Well I seemed to have blown it up by trying to make changes. Oops. I updated to the latest stable build of CP 8.2.9 for the Titano. I rebuilt the project from the code files on Adafruit's website. Changing the name for some of the SSID and Adafruit.IO service through me for a loop but that was a quick name change.

Right now its running connected via Thonny, its working as expected. Hopefully it STAYS that way! :) I will update accordingly. For now..enjoying the artwork!

Respectfully,
Anglerfish27

@anglerfish27
Copy link

So far so good, even got to see some new paintings I had not seeen! I had to disconnect it from Thonny because I needed my laptop so its just running on USB power. So essentially "restart" the timer to see if it crashes or not. Lasted over night, good start!

Cheers,
Anglerfish

@anglerfish27
Copy link

anglerfish27 commented Dec 27, 2023

Sorry for the delay in the update. The fresh build of the latest CP did not help. It crashes with:
HttpError: Code 422: Unprocessable Entity
as well as
ValueError out of bounds

Not sure what to do now :( @FoamyGuy @makermelissa

Respectfully,
Anglerfish27

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants