Truncated image at Google Drive side

[pcbtborges]

Hi, image transfer runs free of errors but some 20% of the images arrive missing part of the footer.
The size of the footer varies from transfer to transfer.
It also happens when I transfer via http to a php script.
Same image works well with FTP transfer.

[cchaves]

I have the same problem, it looks that the image could be too large and the esp32 can't keep up with the connection. I only can get it to work on VGA resolution with quality = 12

[pcbtborges]

Hi, thanks, strange because more than half arrive intact even on higher resolutions.
Will try VGA quality = 12.
Thanks

[chupocro]

The problem is Base64 encoded data encodes every 3 bytes into 4 bytes and after URL encoding when every byte is further encoded into multiple bytes there isn't enough free heap in the RAM to store the whole string to be sent. I am doing experiments with various algorithms for uploading hires pictures to Google Drive for some time and I already can upload complete UXGA (1600x1200) images at quality = 10 but the firmware is still not perfect - I still have to solve a few problems. When I solve the problems I will upload the code to Github.

[michdan61]

Hi chupocro
Have you made any progress with sending UXGA pictures to google drive?
I Can only upload up to SVGA pictures with quality =10.

[kyle-hunady]

@michdan61 I managed to upload UXGA images by collecting the image buffer in two separate Strings and recombining them afterward. I noticed some strange C magic was causing garbage output when using += on a large String; somehow this fixed it.

Here's my repo: https://github.com/kyle-hunady/esp32cam-gdrive
It still has some issues (e.g. first few uploads don't go through), but it might help

[michdan61]

Hello kyle
I would like to see your repo, but The link does not Seem to work.
Could you resend The link.

/michael

[kyle-hunady]

oops, my bad @michdan61 . Should work now. It was set to private

[michdan61]

Thank you Kyle...
By the way, do you know the reason for this problem, why we cant upload bigger images to google drive with the project from gsampallo?
When I use the ES32-cam alone I can take and save pictures in UXGA, so I think the ESP-Cam is able to do it, so where is the problem ?

[kyle-hunady]

You're right, the ESP32 can take pictures okay, but the upload is the problem. I think @chupocro is right about the string being too large.

The program takes the data from the camera and converts it to a String (using base 64 encoding) so that it can be sent to the Google script. The larger the image, the more data, and therefore the larger the String.

Using code similar to gsampallo, I printed the String at different resolutions. The strings are huge for larger images - several thousand characters long. It takes a few seconds for Serial.print() to show it all. But at UXGA, the Google apps script throws an error saying it can't decide the string.

here's the rough pseudocode.for how the image file is constructed as a string

String imageFile = "image/jpeg;base64," // prefix to say that data is an image
for (val in imgBuffer) {
imageFile += base64encode(val)
}

this works okay for smaller images. But on a large image, I noticed something weird - the prefix gets garbled

e.g. after looping, imageFile = "image/jHapJHbV64,HVLPPyVsndoUtNml..."

That shouldn't happen. I assume the String is too big to be doing thousands of += operations. Maybe C++ is freaking out and overwrites the string variable on accident. I can only assume that many other characters are garbled in the huge string.

So I split the loop into two different loops - one for the first half of the buffer, and another for the second half. This way I can make the String smaller, meaning there are less += operations on large strings. I add these strings together afterward, and that seems to work.

I am using code pulled from electronic clinic (https://www.electroniclinic.com/esp32-cam-send-images-to-google-drive-iot-security-camera/), but I believe gsampallo's works the same way

[kyle-hunady]

@michdan61 I did more testing; my initial solution does not actually fix the problem. I provided a real fix at my repo

Here's the process: during my initial testing, I was taking pictures of the ceiling, and all was good. However, when I tried to take a picture of the rest of my room, it would fail. I made sure it wasn't any loose connections.

I took a look at the sizes of the strings generated for these different images. My ceiling was in the range of 70k characters, while my room was near 200k! JPEG compression works great for monochrome images like my ceiling, but is less effective in detailed environments like the rest of my room. So finding enough memory to store that image string was still a problem.

I believe the ESP32 has around 200kB of heap space available for all runtime variables. That leaves probably something like 150kB for my string variable on its own. While my ceiling picture (70k) easily fits, my room picture (200k) could not, causing it to send an overflowed string that could not be decoded by the GApps script.

My old code collected the data in batches, concatenated them together, and sent it all at once. In the end, this process generates a huge string that cannot be stored.

My new code now sends data in batches. This means I only need to store a fraction of the data at a time. It uploads UXGA consistently. :)

[michdan61]

Sounds great, thank you.
I will have a look later on and see how it works...