Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speedup MVT column parser #286

Closed
wants to merge 4 commits into
base: svn-trunk
from

Conversation

Projects
None yet
1 participant
@Algunenano
Copy link
Member

Algunenano commented Aug 10, 2018

Speedup MVT columns parsing by:

  • Avoid multiple strlen when parsing JSON columns.
  • Use heap_deform_tuple instead of GetAttributeByNum for normal columns.
  • Cache the table index and oid for columns to avoid recalculation. Reuse the arrays for Datum and null values instead of allocating and deallocating them for each row. Same for the TupleDesc.

I'm attaching some bechmarks for a dataset where the previous process was a big issue. It has one geometry column (already prepared for MVT), a couple of strings and the rest of the columns (~640) are doubles:

$ cat mc.pgb 
-- Run with something like:
-- pgbench -c 4 -T 30 -f mc.pgb -U postgres mc_perf

SELECT St_AsMVT(q) FROM (
SELECT * FROM tilertest.tract9double
) q;
  • Before:
$ pgbench -c 4 -T 30 -f mc.pgb -U postgres mc_perf
starting vacuum...ERROR:  relation "pgbench_branches" does not exist
(ignoring this error and continuing anyway)
ERROR:  relation "pgbench_tellers" does not exist
(ignoring this error and continuing anyway)
ERROR:  relation "pgbench_history" does not exist
(ignoring this error and continuing anyway)
end.
transaction type: mc.pgb
scaling factor: 1
query mode: simple
number of clients: 4
number of threads: 1
duration: 30 s
number of transactions actually processed: 68
latency average = 1870.297 ms
tps = 2.138698 (including connections establishing)
tps = 2.138903 (excluding connections establishing)
  • After:
$ pgbench -c 4 -T 30 -f mc.pgb -U postgres mc_perf
starting vacuum...ERROR:  relation "pgbench_branches" does not exist
(ignoring this error and continuing anyway)
ERROR:  relation "pgbench_tellers" does not exist
(ignoring this error and continuing anyway)
ERROR:  relation "pgbench_history" does not exist
(ignoring this error and continuing anyway)
end.
transaction type: mc.pgb
scaling factor: 1
query mode: simple
number of clients: 4
number of threads: 1
duration: 30 s
number of transactions actually processed: 806
latency average = 149.464 ms
tps = 26.762339 (including connections establishing)
tps = 26.765364 (excluding connections establishing)

@Algunenano Algunenano force-pushed the Algunenano:trunk_mvt_parse branch from f2f4369 to 1472be4 Aug 10, 2018

@Algunenano

This comment has been minimized.

Copy link
Member Author

Algunenano commented Aug 10, 2018

Travis is having some issues with its network and some builds are failing, but since 3/4 went ok I'll proceed anyway.

@strk strk closed this in 3c81bcc Aug 10, 2018

Algunenano added a commit to Algunenano/postgis that referenced this pull request Aug 13, 2018

Speedup MVT column parsing
Closes #4145
Closes postgis#286

git-svn-id: http://svn.osgeo.org/postgis/trunk@16685 b70326c6-7e19-0410-871a-916f4a2858ee
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.