queryApi.QueryAsync returns GZipped request

[sunserg]

It seems that when Gzip compression is enabled, the responce is not unzipped.

Steps to reproduce:
List the minimal actions needed to reproduce the behavior.

1. Connect to Influx and enable GZip:

 var client = InfluxDBClientFactory.Create(options).EnableGzip();
 var queryApi = client.GetQueryApi();
 var query = $"from(bucket:\"{_bucket}\") |> range(start: {fromDate:O}) |> last()";
 var tables = await _queryApi.QueryAsync(query, _org, cancellationToken);

Expected behavior:
List<FluxTables?> returned.

Actual behavior:
Exception:

CsvHelper.BadDataException: You can ignore bad data by setting BadDataFound to null.
IReader state:
   ColumnCount: 0
   CurrentIndex: -1
   HeaderRecord:

IParser state:
   ByteCount: 0
   CharCount: 202
   Row: 1
   RawRow: 1
   Count: 3
   RawRecord:
▼????k?J►??♠?§??????t{??S %M    ?"T?6♠?Nm)?????M☼?:HB?gw?????q???????|?▼wn:↔w?]▼?0????t???g?????}??/?????<??0]????☼_⌂??W7c?♫?4??\.oO??+?▼???_???L??☼e?,?r(?????P??0-o??♫??~,ô?_?<????^9?????>???=??


   at InfluxDB.Client.Core.Internal.AbstractQueryClient.<>c.<.cctor>b__26_1(Exception e)
   at InfluxDB.Client.Core.Internal.AbstractQueryClient.Query(Func`2 queryFn, Action`1 consumer, Action`1 onError, Action onComplete, CancellationToken cancellationToken)
   at InfluxDB.Client.QueryApi.QueryAsync(Query query, String org, CancellationToken cancellationToken)
...

Specifications:

• Client Version: 4.1.0
• InfluxDB Version: 2.0.7
• Platform: Client: Windows, server: Linux

[bednar]

Hi @sunserg,

thanks for using our client.

I am not able to simulate your BadDataException. I've used following code and everything works well:

using System;
using System.Threading.Tasks;
using InfluxDB.Client;

namespace Examples
{
    public static class GZIPQueryWriteExample
    {
        private const string Host = "http://localhost:9999";
        private const string Token = "my-token";
        private const string Bucket = "my-bucket";
        private const string Organization = "my-org";

        public static async Task Main(string[] args)
        {
            //
            // Initialize Client
            //
            var options = new InfluxDBClientOptions.Builder()
                .Url(Host)
                .AuthenticateToken(Token.ToCharArray())
                .Org(Organization)
                .Bucket(Bucket)
                .Build();
            var client = InfluxDBClientFactory.Create(options).EnableGzip();

            //
            // Prepare data
            //
            await client.GetWriteApiAsync().WriteRecordAsync("mem,tag=a value=1i");

            //
            // Query Data
            //
            var query = $"from(bucket:\"{Bucket}\") |> range(start: 0) |> last()";
            var list = await client.GetQueryApi().QueryAsync(query);
            list.ForEach(it =>
            {
                foreach (var record in it.Records)
                {
                    Console.WriteLine($"{record.GetValueByKey("_time")} {record.GetMeasurement()}: {record.GetField()} {record.GetValue()}");
                }
            });

            client.Dispose();
        }
    }
}

Can you share debug output from the client? You can enable debug mode by:

client.SetLogLevel(LogLevel.Body);

Specifications:

• Client Version: 4.1.0
• InfluxDB Version: 2.0.7
• Platform: Client: MacOS, server: Linux

Regards

[sunserg]

Hi Jakub,

Thank you for looking into it.

This is relevant portion of the log:

...
'Influx2Db.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\6.0.4\System.Runtime.Numerics.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
--> Post /api/v2/query
--> Query: org=allseas
--> Header: Accept-Encoding=application/json
--> Header: Accept=application/json
--> Header: Authorization=Token -.......==
--> Body: {"query":"\nt0 = 2022-03-11T17:33:51.0000000Z\nt1 = 2022-03-11T17:34:51.0000000Z\nperiod = 1s\n\nt_v = from(bucket: \"lor\")\n  |> range(start: t0, stop: t1)\n  |> filter(fn: (r) => (r._measurement == \"LO-331R3-UDP-001\"))\n  |> filter(fn: (r) => (r._field == \"%_of_SWL_AW\"))\n  |> filter(fn: (r) => (r.dev_name == \"ROOT\"))\n  |> aggregateWindow(every: period, fn: mean)\n  |> drop(columns: [\"_start\", \"_stop\"])\n\nt = t_v\n  |> map(fn: (r) => ({ r with quality: 1 }))\n\nt\n  |> rename(columns: { _value: \"value\"})\n  |> set(key: \"tag_id\", value: \"34171\")\n","dialect":{"annotations":["datatype","group","default"],"dateTimeFormat":"RFC3339","header":true,"delimiter":",","commentPrefix":"#"}}
--> END
-->
'Influx2Db.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\6.0.4\System.IO.Compression.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'Influx2Db.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\6.0.4\System.Web.HttpUtility.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'Influx2Db.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\6.0.4\System.Net.Sockets.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'Influx2Db.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\6.0.4\System.Net.NameResolution.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
The thread 0x88bc has exited with code 0 (0x0).
<-- 200
<-- Header: Vary=Accept-Encoding
<-- Header: Date=Wed, 20 Apr 2022 06:53:33 GMT
<-- Header: Transfer-Encoding=chunked
<-- Body: ���<-- END
'Influx2Db.exe' (CoreCLR: clrhost): Loaded 'C:\Users\SSS\source\repos\DLG_datalogger_plugins\Influx2Db\bin\Debug\net6.0\CsvHelper.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
Exception thrown: 'CsvHelper.MissingFieldException' in System.Private.CoreLib.dll
Exception thrown: 'CsvHelper.MissingFieldException' in System.Private.CoreLib.dll
Exception thrown: 'CsvHelper.MissingFieldException' in System.Private.CoreLib.dll
Exception thrown: 'CsvHelper.MissingFieldException' in System.Private.CoreLib.dll
'Influx2Db.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\6.0.4\System.Diagnostics.StackTrace.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
...

Regards, Serge

[bednar]

Thanks for your information, it looks like encoding problem.

Does QueryApi works correctly if you disable GZIP?

[sunserg]

yes

[bednar]

What is your system encoding for the Windows?

[sunserg]

Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.

Try the new cross-platform PowerShell https://aka.ms/pscore6

Loading personal and system profiles took 1252ms.
C:\Users\SSS> [System.Text.Encoding]::Default

IsSingleByte : True
BodyName : iso-8859-1
EncodingName : Western European (Windows)
HeaderName : Windows-1252
WebName : Windows-1252
WindowsCodePage : 1252
IsBrowserDisplay : True
IsBrowserSave : True
IsMailNewsDisplay : True
IsMailNewsSave : True
EncoderFallback : System.Text.InternalEncoderBestFitFallback
DecoderFallback : System.Text.InternalDecoderBestFitFallback
IsReadOnly : True
CodePage : 1252

[bednar]

@sunserg I've used same configuration and GZIPQueryWriteExample works fine.

Which version of dotnet do you use? How do you build your app? Does GZIPQueryWriteExample work for you?

[sunserg]

@bednar
Thank you for looking for a solution.

I'm using .net6.0.4 and .net5.0.16.
Build in VisualStudio 2022.
The issue only exists in v. 4.x.x of the InfluxDB.Client. (v.3.3.0 works)

GZIPQueryWriteExample does work with only one point returned.

However: if I change (to add more data) in the line:

            // Prepare data
            //
           await client.GetWriteApiAsync().WriteRecordAsync("mem,tag=a value=1i");

to

            // Prepare data
            //
            for (int i = 0; i < 1000; i++)
            {
                await client.GetWriteApiAsync().WriteRecordAsync($"mem{i},tag=a value={i}i");
            }

It does not work anymore. Error is the same:

<-- 200
<-- Header: Vary=Accept-Encoding
<-- Header: Date=Fri, 22 Apr 2022 11:31:39 GMT
<-- Header: Transfer-Encoding=chunked
<-- Body: �����8u�v&�Ϯd�xi��k��!R��ޯү�x�c/>�8��9e3�����8uHK���H�6^���7u($�R=��i�m<�yS�2#�g�=m����5o�g����?fT���Eo*�$�7�g{tw�y�Lwd1�u*��S�`�����i位���S�0u��{�k��T\d��e�y..�^�{�{͝�Dfe:�)5ê����S�d�p>������#�ˏtJK+;�fX���5wjPȒI&���6��N
J����O��Vm�m͝����c��8ê����S��J�����F���ۚ;u@J��9Sz�yo#ok���Ճ3��l�U�y[s����$��Ͱj#o��T])E���Ӭgٚ;u R�2<�l"yo$ok�ԁT��>���佑���S�
��Գ�毣��ƭ�p`t̼_���佑���S��gR�;��V��m͝:��U�]�L$��m͝:p	:�
��佑���S�a�d�;�fX5��5w� �����O$��m͝:H��1���D��H��ݩ����h�|��>j_s��ʩ�xW���{#y[s��j���g�7��7��5w�`���$�D��H��ܩ�9��}����){��^~XV$2���H���ۚ;upeV�<W��佑���S���v�ۍ3�����<u��b��#�a�H���C�Q(=�?ê����S�x���)6ê��/�S�r���l6�v���HF���h�佑���S��3��:gl"yo$�k��!�)�R?�H�����<u("�L��fX5��5x�P��)tzQ���m����PG�[kfU�y_s�����z�V��{#y_s���QV��řU��}͝����<{��佑����v�M�����>��.>���6U(I��[v9]|���5n�P�����2]|���5o�P�&�x����g��\�����|���5�z��yS��dV��?�y� �5oj���A���4���4׼i�����������k+?�l��\8픚aՎ+׼����W�i�/���#�����W�� ��I��(u
�����⽼|=�y����S�������l��{7s
��p���=e3����k��@�<�~�*����b͛��c�ˍ��%��&X����B��0~M��k��@Y���D����6>׼i^��Z�جե��������}�2m|���5sz_3�Rςq��l��k�����t�=�3�����:50��x6���g��\C��V�N��Y9q|6��5tj��h��q����8>�ԩ�����)=q|6��5uj�Ɉ~ߪ���g��\S���!�u�����8>�ԩAXY�}���W�t�l�S4� �������B{�T�~w��珙ToP��S�($M�S��[k/:��Q����Y�Cq
����8V�cȳX���85�4���#�Y�Cq
�����0�)��X���85(g��
b�X��85�bQ�O7k��j��k����Y�G�R�H�5o�F�(z�X�b�Ŷ�����≷�gN���Eo*N���?��eN�rj͛:�b���״��6�ּ���K<��-���;���p`�b��y��j�_k���[Y�%o��TM�k��:pEj��b3�����7�sѬ����1��m|�qS?}�HrVT5m|���5n� �R^7�������M���οN0N�_m�k��:�Yp�Ӎ3����"7���g��?�.�7I��N�4�ْ�Jq��j�_k���(BK�dմ��6�����%b��=�_G�=�{���Q�7�������N�l������6�����8u�2Լ/Z�i�m|��S�PI�����6�����8u��g�t�Ͱj�_k��!Qʢ��Ǵ��6��ĩC�c�}oOM�_m�kQ�*��K�9�����ĩC�Y��=cS�W��Z3��e����5m|���5q�P��
�uD5m|���5q��Xd�w�'_'�=c[����r�|Cx��j�_k�4<-- END

In my code it is also working with small amount of data returned. But starting from some number of points it fails to unzip.
So the error seems connected to the size of the response body(?)

Regards, Serge

[bednar]

@sunserg thanks for detail info

I have prepared a fixed version within #317. Development version of the client will be available after #317 acceptance, stable version in regular release cycle.

Regards

[sunserg]

@bednar Thanks! I hope it will be accepted soon.