Optimize batch encoder #250
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Optimization is done by creation of "row encoder" structure that can be used to encode one whole row at a time - no need to lookup OID tables every time. Also, some edoc improvements.
This structure could be reused to encode similar rows without doing OID database lookups for each element
davidw
reviewed
Jan 16, 2021
| @@ -48,6 +48,7 @@ init(Batch) when is_list(Batch) -> | |||
| #batch{batch = Batch}. | |||
|
|
|||
| execute(Sock, #batch{batch = Batch, statement = undefined} = State) -> | |||
| %% Each query has it's own statement | |||
There was a problem hiding this comment.
Linguistic nitpick: "its", not "it's". "It's" == "it is"
davidw
approved these changes
Jan 16, 2021
|
I'm not merging yet, because I plan to wait for my colleagues to do some benchmarks before. |
|
Ok, we tried to import some multi-terabyte dump using both "current" and "optimized" version, but haven't noticed any difference, primarily because the import happened to be disk-bound 😄 . At least, it haven't made anything worse. |
|
Is this ready to merge? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Before the only way to encode Erlang term to PostgreSQL binary format was to use
epgsql_wire:encode_parametersthat callsepgsql_binary:encode/3in a loop. The problem is thatepgsql_binary:encode/3does quite a lot ofcodec()database lookups and structure transformations and this process repeats for each parameter.It worked quite ok for normal
equery, because you usually don't send too many parameters to it. But now with introduction ofCOPY .. FROM STDIN WITH (FORMAT binary)we might need to encode a lot of rows of the same structure. So it makes sense to do all the OID table lookups in advance.The same optimization was applied to
execute_batch/3, because we use the same statement for multiple rows.1st commit optimizes COPY, 2nd optimizes
execute_batch