Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix duckdb extension parallel issue #3937

Merged
merged 6 commits into from
Jul 25, 2024
Merged

Conversation

acquamarin
Copy link
Collaborator

Description

Duckdb::queryResult->Fetch() is not thread safe, thus we have to acquire a lock before fetching the result from duckdb.

Copy link

Benchmark Result

Master commit hash: b59cd52b6d28d3ec6fb9b77c7b979145e7926268
Branch commit hash: d9bcd6bbfc827c00496f6073401dbde4c76d0f66

Query Group Query Name Mean Time - Commit (ms) Mean Time - Master (ms) Diff
aggregation q24 672.31 671.71 0.61 (0.09%)
aggregation q28 11830.52 12072.17 -241.65 (-2.00%)
filter q14 150.96 151.85 -0.89 (-0.58%)
filter q15 154.97 156.11 -1.13 (-0.73%)
filter q16 343.03 326.91 16.12 (4.93%)
filter q17 476.34 472.99 3.35 (0.71%)
filter q18 1950.61 1972.22 -21.61 (-1.10%)
fixed_size_expr_evaluator q07 629.72 593.26 36.46 (6.15%)
fixed_size_expr_evaluator q08 875.59 813.08 62.51 (7.69%)
fixed_size_expr_evaluator q09 867.38 813.91 53.48 (6.57%)
fixed_size_expr_evaluator q10 283.81 270.55 13.26 (4.90%)
fixed_size_expr_evaluator q11 277.45 262.53 14.92 (5.68%)
fixed_size_expr_evaluator q12 270.31 263.72 6.59 (2.50%)
fixed_size_expr_evaluator q13 1554.44 1489.31 65.13 (4.37%)
fixed_size_seq_scan q23 146.94 145.59 1.35 (0.93%)
join q31 44.47 51.77 -7.31 (-14.11%)
ldbc_snb_ic q35 3853.83 3754.42 99.41 (2.65%)
ldbc_snb_ic q36 128.96 131.48 -2.52 (-1.91%)
ldbc_snb_is q32 10.35 10.43 -0.08 (-0.80%)
ldbc_snb_is q33 100.74 98.39 2.36 (2.40%)
ldbc_snb_is q34 98.32 90.23 8.10 (8.97%)
multi-rel multi-rel-large-scan 3279.10 3373.48 -94.38 (-2.80%)
multi-rel multi-rel-lookup 53.65 46.06 7.58 (16.46%)
multi-rel multi-rel-small-scan 62.51 61.03 1.48 (2.42%)
order_by q25 160.35 154.00 6.35 (4.12%)
order_by q26 480.98 462.80 18.19 (3.93%)
order_by q27 1439.48 1444.29 -4.81 (-0.33%)
scan_after_filter q01 207.80 200.33 7.47 (3.73%)
scan_after_filter q02 196.21 191.85 4.36 (2.27%)
shortest_path_ldbc100 q39 146.99 150.83 -3.84 (-2.54%)
var_size_expr_evaluator q03 2122.13 2071.88 50.26 (2.43%)
var_size_expr_evaluator q04 2321.09 2278.34 42.74 (1.88%)
var_size_expr_evaluator q05 2762.95 2651.37 111.57 (4.21%)
var_size_expr_evaluator q06 1398.59 1352.93 45.66 (3.37%)
var_size_seq_scan q19 1516.12 1500.24 15.88 (1.06%)
var_size_seq_scan q20 3181.00 3163.14 17.86 (0.56%)
var_size_seq_scan q21 2434.31 2450.90 -16.60 (-0.68%)
var_size_seq_scan q22 133.17 134.43 -1.26 (-0.94%)

Copy link

Benchmark Result

Master commit hash: b59cd52b6d28d3ec6fb9b77c7b979145e7926268
Branch commit hash: 474da39f564fcb5aa65720f820497e709e1a429d

Query Group Query Name Mean Time - Commit (ms) Mean Time - Master (ms) Diff
aggregation q24 673.50 671.71 1.79 (0.27%)
aggregation q28 11681.45 12072.17 -390.73 (-3.24%)
filter q14 150.04 151.85 -1.81 (-1.19%)
filter q15 152.24 156.11 -3.87 (-2.48%)
filter q16 328.40 326.91 1.49 (0.46%)
filter q17 473.25 472.99 0.25 (0.05%)
filter q18 1972.23 1972.22 0.01 (0.00%)
fixed_size_expr_evaluator q07 591.42 593.26 -1.84 (-0.31%)
fixed_size_expr_evaluator q08 813.04 813.08 -0.04 (-0.01%)
fixed_size_expr_evaluator q09 811.62 813.91 -2.29 (-0.28%)
fixed_size_expr_evaluator q10 267.67 270.55 -2.88 (-1.06%)
fixed_size_expr_evaluator q11 260.98 262.53 -1.55 (-0.59%)
fixed_size_expr_evaluator q12 261.11 263.72 -2.61 (-0.99%)
fixed_size_expr_evaluator q13 1487.81 1489.31 -1.50 (-0.10%)
fixed_size_seq_scan q23 146.12 145.59 0.53 (0.37%)
join q31 46.77 51.77 -5.00 (-9.66%)
ldbc_snb_ic q35 3821.63 3754.42 67.21 (1.79%)
ldbc_snb_ic q36 116.68 131.48 -14.79 (-11.25%)
ldbc_snb_is q32 10.68 10.43 0.24 (2.34%)
ldbc_snb_is q33 99.13 98.39 0.74 (0.75%)
ldbc_snb_is q34 98.72 90.23 8.49 (9.41%)
multi-rel multi-rel-large-scan 3313.30 3373.48 -60.18 (-1.78%)
multi-rel multi-rel-lookup 46.41 46.06 0.34 (0.75%)
multi-rel multi-rel-small-scan 69.45 61.03 8.41 (13.78%)
order_by q25 153.91 154.00 -0.09 (-0.06%)
order_by q26 467.98 462.80 5.18 (1.12%)
order_by q27 1445.43 1444.29 1.14 (0.08%)
scan_after_filter q01 199.52 200.33 -0.81 (-0.41%)
scan_after_filter q02 188.59 191.85 -3.25 (-1.70%)
shortest_path_ldbc100 q39 149.20 150.83 -1.63 (-1.08%)
var_size_expr_evaluator q03 2066.50 2071.88 -5.38 (-0.26%)
var_size_expr_evaluator q04 2275.84 2278.34 -2.50 (-0.11%)
var_size_expr_evaluator q05 2643.40 2651.37 -7.97 (-0.30%)
var_size_expr_evaluator q06 1350.92 1352.93 -2.01 (-0.15%)
var_size_seq_scan q19 1489.22 1500.24 -11.02 (-0.73%)
var_size_seq_scan q20 3174.00 3163.14 10.86 (0.34%)
var_size_seq_scan q21 2442.69 2450.90 -8.22 (-0.34%)
var_size_seq_scan q22 135.58 134.43 1.14 (0.85%)

@acquamarin acquamarin merged commit 2311215 into master Jul 25, 2024
4 of 14 checks passed
@acquamarin acquamarin deleted the duckdb-extension-parallel-fix branch July 25, 2024 06:44
acquamarin added a commit that referenced this pull request Jul 25, 2024
Fix duckdb extension parallel scan issue
ted-wq-x pushed a commit to ted-wq-x/kuzu that referenced this pull request Nov 14, 2024
Fix duckdb extension parallel scan issue

(cherry picked from commit 2311215)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants