issue-29: Downloading files larger than 1Gb causes errors

[wiitt]

Jira ticket: DMAS-82830
matlab.net.http.RequestMessage runs into a limitation when it comes to large amounts of data. At the stage of decoding a response, it produces an error if the response has more than 2^30 elements (1Gb).

Running something like:

params = struct(...             
    'deviceCode', 'ASLCORAS300KHZMAIN',...             
    'dateFrom', '2025-01-12T00:00:00.000Z',...             
    'dateTo', '2025-01-13T00:00:00.000Z',...
    'extension', 'dat'...
    ); 
onc.getDirectFiles(params)

Results in the following:

Error using matlab.net.http.RequestMessage/sendOneRequest
An error occurred while trying to convert the payload of the response from
https://data.oceannetworks.ca/api/archivefiles?token=[YOUR_TOKEN]&method=getFile&filename=ASLCORAS300KHZMAIN_20250112T001422.000Z.dat
using the media-type text. The unconverted data may be found in this exceptions's
History.Response(end).Body. To disable automatic conversion, set HTTPOptions.ConvertResponse to false.

Error in matlab.net.http.RequestMessage/sendAfterChallenge (line 1674)
                [response, request, history] = obj.sendOneRequest(connector, options, ...

Error in matlab.net.http.RequestMessage/sendAndAuthenticate (line 1133)
                                   obj.sendAfterChallenge(response, connector, ...

Error in matlab.net.http.RequestMessage/send (line 543)
                             obj.sendAndAuthenticate([], completedURI, [], connector, options, ...

Error in util/do_request (line 28)
    response = request.send(uri,options);

Error in onc/OncArchive/getFile (line 65)
                util.do_request(url, filters, 'timeout', this.timeout, 'showInfo', this.showInfo, ...

Error in onc/OncArchive/getDirectFiles (line 161)
                    downInfo = this.getFile(filename, overwrite, 'showMsg', false);

Error in matlabClientLibError (line 8)
onc.getDirectFiles(params)

Caused by:
    Error using native2unicode
    Input must contain fewer than 2^30 elements.
Error using native2unicode
Input must contain fewer than 2^30 elements.

Error in matlab.net.http.internal.decodeByteArray>decodeTextStream (line 104)
    data = native2unicode(byteArray', charset);

Error in matlab.net.http.internal.decodeByteArray>dataStreamDecoder/textDecoder (line 95)
            data = string(decodeTextStream(bytes, charset));

Error in matlab.net.http.internal.decodeByteArray (line 27)
        data = decoder(byteArray);

Error in matlab.net.http.internal.readContentFromWebService (line 229)
                data = matlab.net.http.internal.decodeByteArray(payload, charset, ...

Error in matlab.net.http.RequestMessage/sendOneRequest (line 1372)
                            readContentFromWebService(connector, 0, ...

Error in matlab.net.http.RequestMessage/sendAfterChallenge (line 1674)
                [response, request, history] = obj.sendOneRequest(connector, options, ...

Error in matlab.net.http.RequestMessage/sendAndAuthenticate (line 1133)
                                   obj.sendAfterChallenge(response, connector, ...

Error in matlab.net.http.RequestMessage/send (line 543)
                             obj.sendAndAuthenticate([], completedURI, [], connector, options, ...

Error in util/do_request (line 28)
    response = request.send(uri,options);

Error in onc/OncArchive/getFile (line 65)
                util.do_request(url, filters, 'timeout', this.timeout, 'showInfo', this.showInfo, ...

Error in onc/OncArchive/getDirectFiles (line 161)
                    downInfo = this.getFile(filename, overwrite, 'showMsg', false);

Error in matlabClientLibError (line 8)
onc.getDirectFiles(params)