New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

out of memory error when trying to upload blob to iothub #80

Closed
andrewwylie opened this Issue Oct 17, 2017 · 17 comments

Comments

Projects
None yet
9 participants
@andrewwylie

andrewwylie commented Oct 17, 2017

  • OS and version used: OSX High Sierra

  • Python runtime used: 3.6.3

  • SDK version used: latest

Description of the issue:

When running the device sample (device/samples/iothub_client_sample_class.py) the upload fails with the error below, reporting a curl out of memory failure

Code sample exhibiting the issue:

iothub_client_sample_class.py from sdk

Console log of the issue:

python3 iothub_client_sample_class.py -p mqtt -c "HostName=xx;DeviceId=xx"
Python 3.6.3 (default, Oct 12 2017, 14:23:48)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.37)]

IoT Hub Client for Python
Info: IoT Hub SDK for C, version 1.1.25
Starting the IoT Hub Python sample using protocol MQTT...
IoTHubClient sending 5 messages
Error: Time:Tue Oct 17 09:48:49 2017 File:/Users/andrewwylie/Workspace/repos/azure-iot-sdk-python/c/c-utility/adapters/httpapi_curl.c Func:HTTPAPI_ExecuteRequest Line:552 curl_easy_perform() failed: Out of memory

Error: Time:Tue Oct 17 09:48:49 2017 File:/Users/andrewwylie/Workspace/repos/azure-iot-sdk-python/c/c-utility/adapters/httpapi_curl.c Func:HTTPAPI_ExecuteRequest Line:554 (result = HTTPAPI_OPEN_REQUEST_FAILED)
Error: Time:Tue Oct 17 09:48:49 2017 File:/Users/andrewwylie/Workspace/repos/azure-iot-sdk-python/c/c-utility/src/httpapiex.c Func:HTTPAPIEX_ExecuteRequest Line:475 unable to recover sending to a working state
Error: Time:Tue Oct 17 09:48:49 2017 File:/Users/andrewwylie/Workspace/repos/azure-iot-sdk-python/c/iothub_client/src/iothub_client_ll_uploadtoblob.c Func:IoTHubClient_LL_UploadToBlob_step1and2 Line:396 unable to HTTPAPIEX_SAS_ExecuteRequest
Error: Time:Tue Oct 17 09:48:49 2017 File:/Users/andrewwylie/Workspace/repos/azure-iot-sdk-python/c/iothub_client/src/iothub_client_ll_uploadtoblob.c Func:IoTHubClient_LL_UploadToBlob_Impl Line:873 error in IoTHubClient_LL_UploadToBlob_step1
Error: Time:Tue Oct 17 09:48:49 2017 File:/Users/andrewwylie/Workspace/repos/azure-iot-sdk-python/c/iothub_client/src/iothub_client.c Func:uploadingThread Line:1783 unable to IoTHubClient_LL_UploadToBlob
Blob upload confirmation[1001] received for message with result = ERROR

@zolvarga zolvarga self-assigned this Oct 17, 2017

@zolvarga

This comment has been minimized.

Contributor

zolvarga commented Oct 17, 2017

@andrewwylie

Thanks for reporting this.
We will take a look at it.

Best Regards,
Zoltan

@zolvarga

This comment has been minimized.

Contributor

zolvarga commented Nov 10, 2017

@andrewwylie

We have a work item to investigate this. According to the current priorities we can get traction on this next month.

Best Regards,
Zoltan

@zolvarga zolvarga closed this Nov 10, 2017

@drhalftone

This comment has been minimized.

drhalftone commented Nov 17, 2017

I can confirm I'm running into the exact same issue in my C code:

Starting /Users/dllau/Developer/build-LAUAzureIoTWidget-Desktop_Qt_5_9_2_clang_64bit-Debug/LAUAzureIoTWidget.app/Contents/MacOS/LAUAzureIoTWidget...
objc[7117]: Class FIFinderSyncExtensionHost is implemented in both /System/Library/PrivateFrameworks/FinderKit.framework/Versions/A/FinderKit (0x7fff8f718a70) and /System/Library/PrivateFrameworks/FileProvider.framework/OverrideBundles/FinderSyncCollaborationFileProviderOverride.bundle/Contents/MacOS/FinderSyncCollaborationFileProviderOverride (0x109628cd8). One of the two will be used. Which one is undefined.
Info: IoT Hub SDK for C, version 1.1.27
Error: Time:Thu Nov 16 23:24:21 2017 File:/Users/dllau/azure-iot-sdk-c/c-utility/adapters/httpapi_curl.c Func:HTTPAPI_ExecuteRequest Line:556 curl_easy_perform() failed: Out of memory

Error: Time:Thu Nov 16 23:24:21 2017 File:/Users/dllau/azure-iot-sdk-c/c-utility/adapters/httpapi_curl.c Func:HTTPAPI_ExecuteRequest Line:558 (result = HTTPAPI_OPEN_REQUEST_FAILED)
Error: Time:Thu Nov 16 23:24:21 2017 File:/Users/dllau/azure-iot-sdk-c/c-utility/src/httpapiex.c Func:HTTPAPIEX_ExecuteRequest Line:475 unable to recover sending to a working state
Error: Time:Thu Nov 16 23:24:21 2017 File:/Users/dllau/azure-iot-sdk-c/iothub_client/src/iothub_client_ll_uploadtoblob.c Func:IoTHubClient_LL_UploadToBlob_step1and2 Line:396 unable to HTTPAPIEX_SAS_ExecuteRequest
Error: Time:Thu Nov 16 23:24:21 2017 File:/Users/dllau/azure-iot-sdk-c/iothub_client/src/iothub_client_ll_uploadtoblob.c Func:IoTHubClient_LL_UploadToBlob_Impl Line:873 error in IoTHubClient_LL_UploadToBlob_step1
/Users/dllau/Developer/build-LAUAzureIoTWidget-Desktop_Qt_5_9_2_clang_64bit-Debug/LAUAzureIoTWidget.app/Contents/MacOS/LAUAzureIoTWidget exited with code 0

@tameraw tameraw reopened this Nov 22, 2017

@tameraw

This comment has been minimized.

Contributor

tameraw commented Nov 22, 2017

This is still a valid issue and will investigate in the C-SDK. Once fixed, we'll update the Python wrapper SDK.

@jongio

This comment has been minimized.

Contributor

jongio commented Nov 22, 2017

In the meantime, you can use the following sample python/node code to upload files to IoT Hub. It uses the REST APIs directly. https://github.com/jonbgallant/azure-iot-rest/blob/master/data-plane/devices/files/file-upload.py

@openedhardware

This comment has been minimized.

openedhardware commented Nov 30, 2017

I am also getting same error on Raspberry Pi Zero / Python2.7:

Python 2.7.13 (default, Jan 19 2017, 14:48:08) 
[GCC 6.3.0 20170124]
IoT Hub Client for Python
2017-11-30 04:34:45.172377::  Starting Azure IoT Hub client tester...
Info: IoT Hub SDK for C, version 1.1.28
Error: Time:Thu Nov 30 04:34:46 2017 File:/home/pi/azure-iot-sdk-python/c/c-utility/adapters/httpapi_curl.c Func:HTTPAPI_ExecuteRequest Line:556 curl_easy_perform() failed: Out of memory

Error: Time:Thu Nov 30 04:34:46 2017 File:/home/pi/azure-iot-sdk-python/c/c-utility/adapters/httpapi_curl.c Func:HTTPAPI_ExecuteRequest Line:558 (result = HTTPAPI_OPEN_REQUEST_FAILED)
Error: Time:Thu Nov 30 04:34:46 2017 File:/home/pi/azure-iot-sdk-python/c/c-utility/src/httpapiex.c Func:HTTPAPIEX_ExecuteRequest Line:475 unable to recover sending to a working state
Error: Time:Thu Nov 30 04:34:46 2017 File:/home/pi/azure-iot-sdk-python/c/iothub_client/src/iothub_client_ll_uploadtoblob.c Func:IoTHubClient_LL_UploadToBlob_step1and2 Line:396 unable to HTTPAPIEX_SAS_ExecuteRequest
Error: Time:Thu Nov 30 04:34:46 2017 File:/home/pi/azure-iot-sdk-python/c/iothub_client/src/iothub_client_ll_uploadtoblob.c Func:IoTHubClient_LL_UploadToBlob_Impl Line:873 error in IoTHubClient_LL_UploadToBlob_step1
Error: Time:Thu Nov 30 04:34:46 2017 File:/home/pi/azure-iot-sdk-python/c/iothub_client/src/iothub_client.c Func:uploadingThread Line:1844 unable to IoTHubClient_LL_UploadToBlob
2017-11-30 04:34:46.082191 Blob(sample.png) upload confirmation[1001] received for message with result = ERROR

+1 for this issue...

@jongio

This comment has been minimized.

Contributor

jongio commented Dec 4, 2017

get_device also doesn't work:

from iothub_service_client import IoTHubRegistryManager, IoTHubRegistryManagerAuthMethod
IOTHUB_CONNECTION_STRING = "HostName=.azure-devices.net;SharedAccessKeyName=iothubowner;SharedAccessKey="
iothub_registry_manager = IoTHubRegistryManager(IOTHUB_CONNECTION_STRING)
iothub_device = iothub_registry_manager.get_device("device1")

Produces this error:

pi@jongpi6-lite:~/azure-iot-sdk-python/device/tests $ python iothub_device_get_device.py
Error: Time:Mon Dec  4 01:23:44 2017 File:/home/pi/azure-iot-sdk-python/c/c-utility/adapters/httpapi_curl.c Func:HTTPAPI_ExecuteRequest Line:556 curl_easy_perform() failed: Out of memory

Error: Time:Mon Dec  4 01:23:44 2017 File:/home/pi/azure-iot-sdk-python/c/c-utility/adapters/httpapi_curl.c Func:HTTPAPI_ExecuteRequest Line:558 (result = HTTPAPI_OPEN_REQUEST_FAILED)
Error: Time:Mon Dec  4 01:23:44 2017 File:/home/pi/azure-iot-sdk-python/c/c-utility/src/httpapiex.c Func:HTTPAPIEX_ExecuteRequest Line:475 unable to recover sending to a working state
Error: Time:Mon Dec  4 01:23:44 2017 File:/home/pi/azure-iot-sdk-python/c/iothub_service_client/src/iothub_registrymanager.c Func:sendHttpRequestCRUD Line:982 HTTPAPIEX_SAS_ExecuteRequest failed
Traceback (most recent call last):
  File "iothub_device_get_device.py", line 8, in <module>
    iothub_device = iothub_registry_manager.get_device("device1")
iothub_service_client.IoTHubRegistryManagerError: IoTHubRegistryManager.get_device, IoTHubRegistryManagerResult.HTTPAPI_ERROR
@jongio

This comment has been minimized.

Contributor

jongio commented Dec 4, 2017

This is a valid workaround that will remove openssl in favor of gnutls.

cd build_all/linux

./setup.sh

sudo apt remove libcurl4-openssl-dev
sudo apt install libcurl4-gnutls-dev

./build.sh
@drhalftone

This comment has been minimized.

drhalftone commented Dec 4, 2017

@jongio

This comment has been minimized.

Contributor

jongio commented Dec 4, 2017

@drhalftone - Apologies, I didn't notice this was for Mac. I will try soon.

@jongio

This comment has been minimized.

Contributor

jongio commented Dec 20, 2017

@drhalftone - I researched a workaround with Mac, but I wasn't able to find one. The fix should be rolled out soon. @zolvarga @jebrando

@zolvarga

This comment has been minimized.

Contributor

zolvarga commented Jan 9, 2018

@andrewwylie

We are still working on the optimization of memory usage in curl. We will let you know if it is done.

Best Regards,
Zoltan

@ewertons

This comment has been minimized.

Contributor

ewertons commented Jan 11, 2018

Hi @andrewwylie,
this was identified as an external issue and the solution is posted on Azure/azure-iot-sdk-c#308.

Would it be possible for you to try that and confirm if it addresses your issue?

@zolvarga

This comment has been minimized.

Contributor

zolvarga commented Jan 18, 2018

Closed as duplicate and answered

@zolvarga zolvarga closed this Jan 18, 2018

@royaza

This comment has been minimized.

royaza commented Feb 26, 2018

@zolvarga Hi, I have the same issue on Mac OS. Is there any update about it?

Invoking device to reboot...
Error: Time:Mon Feb 26 09:42:33 2018 File:/Users/X/azure-iot-sdk-python/c/c-utility/adapters/httpapi_curl.c Func:HTTPAPI_ExecuteRequest Line:587 curl_easy_perform() failed: Out of memory

Error: Time:Mon Feb 26 09:42:33 2018 File:/Users/X/azure-iot-sdk-python/c/c-utility/adapters/httpapi_curl.c Func:HTTPAPI_ExecuteRequest Line:589 (result = HTTPAPI_OPEN_REQUEST_FAILED)
Error: Time:Mon Feb 26 09:42:33 2018 File:/Users/X/azure-iot-sdk-python/c/c-utility/src/httpapiex.c Func:HTTPAPIEX_ExecuteRequest Line:475 unable to recover sending to a working state
Error: Time:Mon Feb 26 09:42:33 2018 File:/Users/X/azure-iot-sdk-python/c/iothub_service_client/src/iothub_devicemethod.c Func:sendHttpRequestDeviceMethod Line:321 HTTPAPIEX_SAS_ExecuteRequest failed
Error: Time:Mon Feb 26 09:42:33 2018 File:/Users/X/azure-iot-sdk-python/c/iothub_service_client/src/iothub_devicemethod.c Func:IoTHubDeviceMethod_Invoke Line:489 Failure sending HTTP request for device method invoke

Thanks a lot,

@ToadTWP666

This comment has been minimized.

ToadTWP666 commented Mar 5, 2018

I am executing the following code on my RPi Model 3. This code executes perfect on my Surface Pro so I know it will work:

import time
import shutil
import sys
import iothub_client
import os
from iothub_client import IoTHubClient, IoTHubClientError, IoTHubTransportProvider, IoTHubClientResult, IoTHubError

CONNECTION_STRING = ""
PROTOCOL = IoTHubTransportProvider.HTTP

source = '/media/pi/Navistar/images' # directory where raw images are stored
dest1 = '/media/pi/Navistar/image_history/' # directory where archived images will be moved

def blob_upload_conf_callback(result, user_context):
if str(result) == 'OK':
print ( "...file uploaded successfully." )
else:
print ( "...file upload callback returned: " + str(result) )

def iothub_file_upload_sample_run():
try:
print ( "IoT Hub file upload sample, press Ctrl-C to exit" )
client = IoTHubClient(CONNECTION_STRING, PROTOCOL)

    for root, dirs, filenames in os.walk(source):
        for f in filenames:
            FILENAME = f
            print(f)
            fullpath = os.path.join(source, f)
            f1 = open(fullpath, 'rb')
            content = f1.read()
            client.upload_blob_async(FILENAME, content, len(content), blob_upload_conf_callback, 0)
            f1.close()
            #shutil.move(source+f, dest1)  # Move all of the files into archive directory


except IoTHubError as iothub_error:
    print ( "Unexpected error %s from IoTHub" % iothub_error )
    return
except KeyboardInterrupt:
    print ( "IoTHubClient sample stopped" )
except:
    print ( "generic error" )

if name == 'main':
print ( "Simulating a file upload using the Azure IoT Hub Device SDK for Python" )
print ( " Protocol %s" % PROTOCOL )
print ( " Connection string=%s" % CONNECTION_STRING )

iothub_file_upload_sample_run()
exit()
@ToadTWP666

This comment has been minimized.

ToadTWP666 commented Mar 5, 2018

When I run it on my RPI Model 3. Running Raspian (stretch) version 9 and python 2.7.13 with I get the following errors:

Time:Mon Mar 5 01:13:27 2018
File:/home/Navistar/azure-iot-sdk-python/c/c-utility/adapters/httpapi_curl.c Func:HTTPAPI_ExecuteRequest Line:587 curl_easy_perform() failed:
Out of memory

Error:
Time:Mon Mar 5 01:13:27 2018 File:/home/Navistar/azure-iot-sdk-python/c/c-utility/adapters/httpapi_curl.c Func:HTTPAPI_ExecuteRequest Line:589 (result = HTTPAPI_OPEN_REQUEST_FAILED)
Error: Time:Mon Mar 5 01:13:27 2018 File:/home/Navistar/azure-iot-sdk-python/c/c-utility/src/httpapiex.c Func:HTTPAPIEX_ExecuteRequest Line:475 unable to recover sending to a working state
Error: Time:Mon Mar 5 01:13:27 2018 File:/home/Navistar/azure-iot-sdk-python/c/iothub_client/src/iothub_client_ll_uploadtoblob.c Func:IoTHubClient_LL_UploadToBlob_step1and2 Line:406 unable to HTTPAPIEX_SAS_ExecuteRequest
Error: Time:Mon Mar 5 01:13:27 2018 File:/home/Navistar/azure-iot-sdk-python/c/iothub_client/src/iothub_client_ll_uploadtoblob.c Func:IoTHubClient_LL_UploadMultipleBlocksToBlob_Impl Line:915 error in IoTHubClient_LL_UploadToBlob_step1
Error: Time:Mon Mar 5 01:13:27 2018 File:/home/Navistar/azure-iot-sdk-python/c/iothub_client/src/iothub_client.c Func:uploadingThread Line:2000 unable to IoTHubClient_LL_UploadToBlob
...file upload callback returned: ERROR
root@raspberrypi:/home/Navistar/azure-iot-sdk-python/device/samples#

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment