while retrieving chunks of data through URL it was throwing an exception #9

CPavanKishore · 2018-01-29T15:07:21Z

while retrieving chunks of data through URL it was throwing an exception ,is this because of continues hitting of URL with out threading......?

Can you please resolve this issue sooner

howryu · 2018-01-29T17:21:36Z

This exception is thrown while retrieving a large chunk of data. Normally, server will write data to a S3 bucket and client will download result from the s3 bucket. This looks like request timed out when trying to download data from s3. Just wondering if your host can connect to s3 or not? Do you have any proxy config?

CPavanKishore · 2018-01-29T17:47:41Z

Yes we have large chunk of data around min of 30k , even we have a proxy firewall it is getting 1st chunk of data and showing it in the console after continuously hitting the rest chunks it is throwing error

CPavanKishore · 2018-01-29T17:47:57Z

Is there any way to resolve this issue....?

howryu · 2018-01-29T17:57:54Z

@CPavanKishore First chunk of data is returned from snowflake server directly. So basically, the first response for a large result set contains result set metadata, a few rows of data, and the urls to the rest of data in s3 bucket. So my question to you is: are you able to connect s3 from your dev box? A simple curl (or whatever cmd tool on windows) command can verify if you are able to connect to s3.

CPavanKishore · 2018-01-30T05:40:49Z

i'm able to connect s3 bucket and get the data

i tried all the chunk URL's through postman @howryu

Check in the above picture

howryu · 2018-01-30T17:34:52Z

Hmm. Ok I need to look into it since this cannot reproduced in my environment.

Can you try to enable logging and pass me the log file? I have some docs in https://github.com/snowflakedb/snowflake-connector-net/blob/master/README.md

CPavanKishore · 2018-01-31T07:55:28Z

Application Insights Telemetry (unconfigured): {"name":"Microsoft.ApplicationInsights.Dev.RemoteDependency","time":"2018-01-31T07:52:46.0413218Z","tags":{"ai.cloud.roleInstance":"pkreddy-T440","ai.location.ip":"::1","ai.internal.nodeName":"pkreddy-T440","ai.internal.sdkVersion":"rdddsc:2.4.1-1362","ai.operation.parentId":"|e17a9f54-413f5b2c42453c9b.","ai.operation.id":"e17a9f54-413f5b2c42453c9b","ai.operation.name":"GET Values/GetAsync","ai.application.ver":"1.0.0.0"},"data":{"baseType":"RemoteDependencyData","baseData":{"ver":2,"name":"POST /session/v1/login-request","id":"|e17a9f54-413f5b2c42453c9b.1.","data":"https://juniperdev.snowflakecomputing.com:443/session/v1/login-request?warehouse=EDP_WH&databaseName=EDP_DB&schemaName=&requestId=7aa6857c-deaa-46e2-9479-c0df8bbf7fad","duration":"00:00:02.0985810","resultCode":"200","success":true,"type":"Http","target":"juniperdev.snowflakecomputing.com","properties":{"AspNetCoreEnvironment":"Development","DeveloperMode":"true"}}}}
Application Insights Telemetry (unconfigured): {"name":"Microsoft.ApplicationInsights.Dev.RemoteDependency","time":"2018-01-31T07:52:46.7594122Z","tags":{"ai.cloud.roleInstance":"pkreddy-T440","ai.location.ip":"::1","ai.internal.nodeName":"pkreddy-T440","ai.internal.sdkVersion":"rdddsc:2.4.1-1362","ai.operation.parentId":"|e17a9f54-413f5b2c42453c9b.","ai.operation.id":"e17a9f54-413f5b2c42453c9b","ai.operation.name":"GET Values/GetAsync","ai.application.ver":"1.0.0.0"},"data":{"baseType":"RemoteDependencyData","baseData":{"ver":2,"name":"POST /queries/v1/query-request","id":"|e17a9f54-413f5b2c42453c9b.2.","data":"https://juniperdev.snowflakecomputing.com:443/queries/v1/query-request?requestId=7297efe3-d13b-45c0-b8e6-ff80c4710139","duration":"00:00:00.5269655","resultCode":"200","success":true,"type":"Http","target":"juniperdev.snowflakecomputing.com","properties":{"AspNetCoreEnvironment":"Development","DeveloperMode":"true"}}}}
Time Elapsed in Custom : 4314.7858'dotnet.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.0.3\System.Core.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'dotnet.exe' (CoreCLR: clrhost): Loaded 'cxni12yu.hni'. Cannot find or open the PDB file.
'dotnet.exe' (CoreCLR: clrhost): Loaded 'hygaco1y.bm5'. Cannot find or open the PDB file.
Application Insights Telemetry (unconfigured): {"name":"Microsoft.ApplicationInsights.Dev.Message","time":"2018-01-31T07:52:48.2592901Z","tags":{"ai.cloud.roleInstance":"pkreddy-T440","ai.location.ip":"::1","ai.internal.nodeName":"pkreddy-T440","ai.internal.sdkVersion":"aspnet5c:2.1.1","ai.operation.parentId":"|e17a9f53-413f5b2c42453c9b.","ai.operation.id":"e17a9f53-413f5b2c42453c9b","ai.operation.name":"GET Home/Index","ai.application.ver":"1.0.0.0"},"data":{"baseType":"MessageData","baseData":{"ver":2,"message":"Executing ViewResult, running view at path /Views/Home/Index.cshtml.","severityLevel":"Information","properties":{"Path":"/Views/Home/Index.cshtml","CategoryName":"Microsoft.AspNetCore.Mvc.ViewFeatures.Internal.ViewResultExecutor","AspNetCoreEnvironment":"Development","DeveloperMode":"true","{OriginalFormat}":"Executing ViewResult, running view at path {Path}."}}}}
'dotnet.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\store\x64\netcoreapp2.0\microsoft.aspnetcore.webutilities\2.0.1\lib\netstandard2.0\Microsoft.AspNetCore.WebUtilities.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
Application Insights Telemetry (unconfigured): {"name":"Microsoft.ApplicationInsights.Dev.RemoteDependency","time":"2018-01-31T07:52:48.7391938Z","tags":{"ai.cloud.roleInstance":"pkreddy-T440","ai.location.ip":"::1","ai.internal.nodeName":"pkreddy-T440","ai.internal.sdkVersion":"rdddsc:2.4.1-1362","ai.operation.parentId":"|e17a9f54-413f5b2c42453c9b.","ai.operation.id":"e17a9f54-413f5b2c42453c9b","ai.operation.name":"GET Values/GetAsync","ai.application.ver":"1.0.0.0"},"data":{"baseType":"RemoteDependencyData","baseData":{"ver":2,"name":"GET /mgz0-s-ssca0741/results/4bca04e6-5be3-47cf-acaf-2aeda65c2b01/main/data_0_0_1","id":"|e17a9f54-413f5b2c42453c9b.3.","data":"https://sfc-ds1-customer-stage.s3.amazonaws.com:443/mgz0-s-ssca0741/results/4bca04e6-5be3-47cf-acaf-2aeda65c2b01/main/data_0_0_1?x-amz-server-side-encryption-customer-algorithm=AES256&response-content-encoding=gzip&AWSAccessKeyId=AKIAITDIMWYCMJHC6WAQ&Expires=1517406766&Signature=aHGT%2BfduoE9z9%2BR%2B9fyr4%2BLibYo%3D","duration":"00:00:01.7588694","resultCode":"200","success":true,"type":"Http","target":"sfc-ds1-customer-stage.s3.amazonaws.com","properties":{"AspNetCoreEnvironment":"Development","DeveloperMode":"true"}}}}
'dotnet.exe' (CoreCLR: clrhost): Loaded 'gb0zvkbd.mmk'. Cannot find or open the PDB file.
Application Insights Telemetry (unconfigured): {"name":"Microsoft.ApplicationInsights.Dev.RemoteDependency","time":"2018-01-31T07:52:48.8229514Z","tags":{"ai.cloud.roleInstance":"pkreddy-T440","ai.location.ip":"::1","ai.internal.nodeName":"pkreddy-T440","ai.internal.sdkVersion":"rdddsc:2.4.1-1362","ai.operation.parentId":"|e17a9f54-413f5b2c42453c9b.","ai.operation.id":"e17a9f54-413f5b2c42453c9b","ai.operation.name":"GET Values/GetAsync","ai.application.ver":"1.0.0.0"},"data":{"baseType":"RemoteDependencyData","baseData":{"ver":2,"name":"GET /mgz0-s-ssca0741/results/4bca04e6-5be3-47cf-acaf-2aeda65c2b01/main/data_0_0_0","id":"|e17a9f54-413f5b2c42453c9b.4.","data":"https://sfc-ds1-customer-stage.s3.amazonaws.com:443/mgz0-s-ssca0741/results/4bca04e6-5be3-47cf-acaf-2aeda65c2b01/main/data_0_0_0?x-amz-server-side-encryption-customer-algorithm=AES256&response-content-encoding=gzip&AWSAccessKeyId=AKIAITDIMWYCMJHC6WAQ&Expires=1517406766&Signature=DjCN5ct46udjvcKNEFdgU8CiVhw%3D","duration":"00:00:01.8426168","resultCode":"200","success":true,"type":"Http","target":"sfc-ds1-customer-stage.s3.amazonaws.com","properties":{"AspNetCoreEnvironment":"Development","DeveloperMode":"true"}}}}
'dotnet.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.0.3\Microsoft.CSharp.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'dotnet.exe' (CoreCLR: clrhost): Loaded 'u1zvlzv1.ln0'. Cannot find or open the PDB file.
'dotnet.exe' (CoreCLR: clrhost): Loaded '2bcurmqa.2in'. Cannot find or open the PDB file.
'dotnet.exe' (CoreCLR: clrhost): Loaded '3vpp3n2o.jbq'. Cannot find or open the PDB file.
Application Insights Telemetry (unconfigured): {"name":"Microsoft.ApplicationInsights.Dev.Message","time":"2018-01-31T07:52:49.9292117Z","tags":{"ai.cloud.roleInstance":"pkreddy-T440","ai.location.ip":"::1","ai.internal.nodeName":"pkreddy-T440","ai.internal.sdkVersion":"aspnet5c:2.1.1","ai.operation.parentId":"|e17a9f53-413f5b2c42453c9b.","ai.operation.id":"e17a9f53-413f5b2c42453c9b","ai.operation.name":"GET Home/Index","ai.application.ver":"1.0.0.0"},"data":{"baseType":"MessageData","baseData":{"ver":2,"message":"Executed action LegalAnalyticsPlatform.Controllers.HomeController.Index (LegalAnalyticsPlatform) in 8922.2328ms","severityLevel":"Information","properties":{"CategoryName":"Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker","AspNetCoreEnvironment":"Development","DeveloperMode":"true","ElapsedMilliseconds":"8922.2328","{OriginalFormat}":"Executed action {ActionName} in {ElapsedMilliseconds}ms","ActionName":"LegalAnalyticsPlatform.Controllers.HomeController.Index (LegalAnalyticsPlatform)"}}}}
Application Insights Telemetry (unconfigured): {"name":"Microsoft.ApplicationInsights.Dev.Message","time":"2018-01-31T07:52:49.9392628Z","tags":{"ai.cloud.roleInstance":"pkreddy-T440","ai.location.ip":"::1","ai.internal.nodeName":"pkreddy-T440","ai.internal.sdkVersion":"aspnet5c:2.1.1","ai.operation.parentId":"|e17a9f53-413f5b2c42453c9b.","ai.operation.id":"e17a9f53-413f5b2c42453c9b","ai.operation.name":"GET Home/Index","ai.application.ver":"1.0.0.0"},"data":{"baseType":"MessageData","baseData":{"ver":2,"message":"Request finished in 9584.2819ms 200 text/html; charset=utf-8","severityLevel":"Information","properties":{"CategoryName":"Microsoft.AspNetCore.Hosting.Internal.WebHost","StatusCode":"200","AspNetCoreEnvironment":"Development","DeveloperMode":"true","ContentType":"text/html; charset=utf-8","ElapsedMilliseconds":"9584.2819"}}}}
Application Insights Telemetry (unconfigured): {"name":"Microsoft.ApplicationInsights.Dev.Request","time":"2018-01-31T07:52:40.3087757Z","tags":{"ai.cloud.roleInstance":"pkreddy-T440","ai.location.ip":"::1","ai.internal.nodeName":"pkreddy-T440","ai.internal.sdkVersion":"aspnet5c:2.1.1","ai.operation.id":"e17a9f53-413f5b2c42453c9b","ai.operation.name":"GET Home/Index","ai.application.ver":"1.0.0.0"},"data":{"baseType":"RequestData","baseData":{"ver":2,"id":"|e17a9f53-413f5b2c42453c9b.","name":"GET Home/Index","duration":"00:00:09.6721309","success":true,"responseCode":"200","url":"http://localhost:50779/","properties":{"AspNetCoreEnvironment":"Development","DeveloperMode":"true","httpMethod":"GET"}}}}
'dotnet.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.0.3\Microsoft.Win32.Primitives.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
An unhandled exception of type 'Snowflake.Data.Client.SnowflakeDbException' occurred in System.Private.CoreLib.dll
External component has thrown an exception.

'dotnet.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.0.3\System.Diagnostics.StackTrace.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
The program '[14096] dotnet.exe' has exited with code 0 (0x0).

is this the one which you asked ..?

howryu · 2018-02-01T02:27:52Z

@CPavanKishore No. I mean the log file generated by .NET Driver.
Here is the example https://github.com/snowflakedb/snowflake-connector-net/blob/master/Snowflake.Data.Tests/App.config on how to config to enable logging.

More importantly, I probably need the network trace to understand why you cannot connect to S3.
https://docs.microsoft.com/en-us/dotnet/framework/network-programming/how-to-configure-network-tracing

The example I provided (App.Config) also includes the network trace. Can you generate the trace to see what's going on?

CPavanKishore · 2018-02-02T09:40:13Z

i tried but common.logging supports up to .netcore1.0 it doesn't support 2.0,Can you try to retrieve a large amount of data from your side to check

howryu · 2018-02-02T17:29:37Z

I compiled the .net driver against .net framework 4.6, but not .net core (there is a plan porting it)
#7
So I cannot make sure if it worked with .net core or not.

I do have a large result tests in
https://github.com/snowflakedb/snowflake-connector-net/blob/master/Snowflake.Data.Tests/SFDbCommandIT.cs
which runs successfully on appveyor.

CPavanKishore · 2018-02-06T15:57:13Z

Can i get any paid session to configure this snowflake connector in my project @howryu ....?

or any alternatives for this ..?

howryu · 2018-03-05T17:56:20Z

This is already fixed in #10. More specifically, in

snowflake-connector-net/Snowflake.Data/Core/SFChunkDownloader.cs

Line 125 in 45d1a0f

timeout = TimeSpan.FromHours(1),

Reason is because the http request timeout is in milliseconds. It should be 1 hour (3600 * 1000 millis) in stead of (3600 millis).

akrock mentioned this issue Mar 2, 2018

Fix async support #15

Closed

sfc-gh-hkapre added the PC_Blocker label Mar 2, 2018

howryu closed this as completed Mar 5, 2018

howryu mentioned this issue Apr 10, 2018

Conn.open() Taking longer and doesn't responding back #5

Closed

oryahud mentioned this issue Feb 10, 2019

Error encountered when downloading a result chunk snowflakedb/snowflake-jdbc#101

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

while retrieving chunks of data through URL it was throwing an exception #9

while retrieving chunks of data through URL it was throwing an exception #9

CPavanKishore commented Jan 29, 2018 •

edited

howryu commented Jan 29, 2018

CPavanKishore commented Jan 29, 2018 •

edited

CPavanKishore commented Jan 29, 2018

howryu commented Jan 29, 2018

CPavanKishore commented Jan 30, 2018

howryu commented Jan 30, 2018

CPavanKishore commented Jan 31, 2018 •

edited

howryu commented Feb 1, 2018

CPavanKishore commented Feb 2, 2018

howryu commented Feb 2, 2018

CPavanKishore commented Feb 6, 2018

howryu commented Mar 5, 2018 •

edited

while retrieving chunks of data through URL it was throwing an exception #9

while retrieving chunks of data through URL it was throwing an exception #9

Comments

CPavanKishore commented Jan 29, 2018 • edited

howryu commented Jan 29, 2018

CPavanKishore commented Jan 29, 2018 • edited

CPavanKishore commented Jan 29, 2018

howryu commented Jan 29, 2018

CPavanKishore commented Jan 30, 2018

howryu commented Jan 30, 2018

CPavanKishore commented Jan 31, 2018 • edited

howryu commented Feb 1, 2018

CPavanKishore commented Feb 2, 2018

howryu commented Feb 2, 2018

CPavanKishore commented Feb 6, 2018

howryu commented Mar 5, 2018 • edited

CPavanKishore commented Jan 29, 2018 •

edited

CPavanKishore commented Jan 29, 2018 •

edited

CPavanKishore commented Jan 31, 2018 •

edited

howryu commented Mar 5, 2018 •

edited