SNOW-857660: Improve JSON performance #880
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sfc-gh-pfus
force-pushed
the
SNOW-857660-improve-json
branch
from
14b3993
to
bb7fbdf
Compare
sfc-gh-pbulawa
approved these changes
Aug 7, 2023
| loc, err = time.LoadLocation(*tz) | ||
| if err != nil { | ||
| loc = time.Now().Location() | ||
| } | ||
| } | ||
| paramsMutex.Unlock() |
There was a problem hiding this comment.
How to take care of ALTER SESSION command changes some parameter like TIMEZONE? will it be change in the cache? Could you please add test case for it?
There was a problem hiding this comment.
You are right. With your question in my mind, I started from scratch: #882
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Before my change, for each row we tried to find out the location for string localisation (like formatting dates etc, even if it was not relevant). Checking location was easy if someone specified the location in the connection parameter, but if it was not (and I believe it is usual), we needed to guess this location. This involved very inefficient system call for each row and each column. I changed it so the location is cached for connection. The code handling JSON responses is like 30-50 times faster (!!!). Quick stats: before my change 1M rows (single column) was retrieved in 1,5min. After my changes it takes 3 seconds. In 1,5min we are able to handle nearly 50M of rows.
Checklist
make fmtto fix inconsistent formatsmake lintto get lint errors and fix all of them