Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Berry: fix file.readbytes() for large files #21657

Closed
wants to merge 1 commit into from

Conversation

Staars
Copy link
Contributor

@Staars Staars commented Jun 19, 2024

Description:

Reading binary files into bytes in Berry leads to data corruption for large files (I did not figure out the threshold) by creating a bytes buffer of the correct size, that has correct data at the front but only holds zeros at the tail end. This fails silently and does not throw any error.

This seems to be related to the resize operation after the actual data read from the file system.

To reproduce we need a test file on the ESP, that is not too small. I used this one:
https://github.com/Staars/MockUp/blob/main/watermeter.jpg

Then load in the Berry web console:

f = open("watermeter.jpg","r")
b =f.readbytes()
f.close()

Check the terminating bytes, which should be "FFD9" for a JPG file with:

b[-2..]

The current version shows "0000". Tested on ESP32 and ESP32-S3.

Unrelated to this PR the C6 crashes with heap corruption:
CORRUPT HEAP: Bad head at 0x408407e8. Expected 0xabba1234 got 0x52844590

Checklist:

  • The pull request is done against the latest development branch
  • Only relevant files were touched
  • Only one feature/fix was added per PR and the code change compiles without warnings
  • The code change is tested and works with Tasmota core ESP8266 V.2.7.6
  • The code change is tested and works with Tasmota core ESP32 V.3.0.1
  • I accept the CLA.

NOTE: The code change must pass CI tests. Your PR cannot be merged unless tests pass

@sfromis
Copy link
Contributor

sfromis commented Jun 19, 2024

Looks like some much wider issue with non-small files. I tried to transfer a moderately sized file (ftp.be) with both Manage File system upload (web or via curl), or the UrlFetch command, on a board with PSRAM available, and this failed. Closest to a symptom was "file too large". I did manage to transfer a somewhat smaller .be file, 11 KB instead of failing with 17 KB.

@s-hadinger
Copy link
Collaborator

I'm afraid there is a deeper bug in the bytes.resize function. Let's put on hold until I see more clearly what is happening

@s-hadinger s-hadinger added the on hold by dev team Result - Feature request put on hold by member of development team label Jun 19, 2024
@s-hadinger
Copy link
Collaborator

Closing this one, the actual fix is #21716

There was a bug in bytes.resize()

@s-hadinger s-hadinger closed this Jul 1, 2024
@Staars
Copy link
Contributor Author

Staars commented Jul 1, 2024

Nice!
(But I still do not really see the value of the second resize procedure.)

@s-hadinger
Copy link
Collaborator

The last resize is only here in case the number of bytes read is different from what was expected. It should not happen with Tasmota, but you never know. There are many reason why a read could return less bytes than the size of the file.

@Jason2866
Copy link
Collaborator

@s-hadinger The crash with the C6 is still there

@Staars Staars deleted the bry_filelib branch July 11, 2024 17:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
on hold by dev team Result - Feature request put on hold by member of development team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants