Release 6.22.1_arenadata41#462
Closed
Stolb27 wants to merge 7 commits into
Closed
Conversation
It repalced the hard-coded define MAX_SCAN_ON_SHMEM. Specifies the limit of maximum scan node's instrumentations per query in shmem. If table has many partitions, Postgres planner will generate a plan with many SCAN nodes under a APPEND node. If the number of partitions are too many, this plan will occupy too many slots. Here is a limitation on number of shmem slots used by scan nodes for each backend. Instruments exceeding the limitation are allocated in local memory.
Steps to reproduce: 1. Create function written in sql language that optimizers can’t inline, e. g.: create function f(a int) returns int immutable security definer language sql as $$ select case when $1 > 500 then 0 end; $$; 2. Execute a query which executes this function multiple times explain analyze verbose select f(i) from generate_series(1, 1000000) i; 3. The query uses much more memory when gp_enable_gpperfmon=true than in the opposite case. The reason of high memory consumption is memory allocation for two structures in CreateQueryDesc() and release of memory for only one of these structures in FreeQueryDesc(). This pair of functions is run every time when subquery is executed. The more times the subquery is executed, the more memory leaks. The patch adds deallocation of the second structure after data in it is no longer needed. There is no leak in _SPI_execute_plan(), but deallocation is added here to be sure that the leak would not appear in the future.
…status_kv functions on entryDB (#390) The pg_resgroup_get_status and pg_resgroup_get_status_kv functions returned invalid values when they was called in insert-select queries. The reason for the error was that the functions was called on entry db only. They should be called on all segments. The error can be fixed by setting the proexeclocation property of the functions to 'i' (PROEXECLOCATION_INITPLAN). But changing catalog for stable release is something we should avoid unless absolutely necessary for critical issues. So the execution of pg_resgroup_get_status and pg_resgroup_get_status_kv functions is disallowed on entryDB. User will get error message and hint how to call the functions. https://github.com/greenplum-db/gpdb/issues/14154
Before this patch while running gpcheckperf uitlity the buffer was set by default for undelying gpnetbenchClient utility as 32Kb. It leaded to problem with receiving of annoying and misleading warnings about connection between hosts. Use '--buffer-size' flag with size in kilobytes to set buffer size, which will be used at gpnetbenchClient. It is optional parameter. Default value is 8Kb.
) gpdb does not truncate error log correctly on error in external web tables. If error code is not zero and error message is truncated in interpretError on multi-byte character, then assert occurs. If error code is zero and command emit stderr and this error message is truncated in write_log (which is called by read_err_msg) on multi-byte character, then bad character writes to log and later this log can not be correctly read. Solution is to correct error log truncation according to database encoding.
Statistic context wasn't destroyed with query descriptor, so queries with
multiple call of sql functions consumed a lot of memory.
Steps to reproduce:
LOAD 'auto_explain';
SET auto_explain.log_nested_statements=true;
SET auto_explain.log_min_duration = 0;
SET auto_explain.log_analyze = true;
explain analyze select information_schema._pg_truetypid(a, t) from (
select a as a, t as t from pg_type t
join pg_attribute a on a.atttypid = t.oid
where typtype = 'd' limit 1
) at join generate_series(1, 9000000) gc on true;
Changes:
1. Add cdbexplain_showStatCtxFree() function to release memory allocated for
statistic context. The function is called from FreeQueryDesc() when the
statistic context exists. To check whether the context exists the assigning NULL
value to the showstatctx field of query descriptor is moved from
ExplainOnePlan() to CreateQueryDesc() and CreateUtilityQueryDesc().
2. Statistic context in auto_explain isn't created if the context already exists.
It may be created in ExplainOnePlan().
3. When memory is allocated in functions called from explain_ExecutorEnd(), the
context of query is used. This memory is released in
standard_ExecutorEnd() -> FreeExecutorState().
…ution for Sequence children (#446) With some conditions ORCA generates Redistribute Motion from all segments to one segment or generates a plan which can't be executed by reason of inconsistency between CTE producers and consumers. In such plans Shared Scan is CTE producer that's executed only on one segment (due to Redistribute Motion from all segments to one segments), but CTE consumers are executed on all segments and then query hangs. We get Redistribute from all segments to one segments because the following. When translating expression to DXL, ORCA sets one segment for input array if strict or tainted replicated distribution detected. This one segment is used as output array for below motions including redistribute and broadcast. Solution is to prohibit replicated distribution when required distribution in sequence is singleton on master or non-singleton with prohibited replicated. In case of inlining one CTE we also need to push down non-singleton with prohibited replicated through filter. Because filter is appeared when CTE may be inlined and inlining allowed or only one consumer exists for that CTE. In this case ORCA append logical select, which later is translated to filter.
Collaborator
Author
|
release branch must be updated first |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes: