ADBDEV-4015 2.2.2 sync by deart2k · Pull Request #23 · arenadata/diskquota

deart2k · 2023-07-25T08:05:02Z

No description provided.

Note: `TextDatumGetCString()` returns a null-terminated CString. ``` [ 12%] Building C object CMakeFiles/diskquota.dir/src/diskquota.c.o In file included from /home/v/workspace/diskquota/src/diskquota.h:16, from /home/v/workspace/diskquota/src/diskquota.c:18: /home/v/workspace/diskquota/src/diskquota.c: In function ‘diskquota_status_schema_version’: /home/v/.local/gpdb7/include/postgresql/server/c.h:957:25: warning: ‘strncpy’ specified bound 64 equals destinatio n size [-Wstringop-truncation] 957 | strncpy(_dst, (src), _len); \ | ^~~~~~~~~~~~~~~~~~~~~~~~~~ /home/v/workspace/diskquota/src/diskquota.c:1548:9: note: in expansion of macro ‘StrNCpy’ 1548 | StrNCpy(version, vv, sizeof(version)); | ^~~~~~~ ``` Co-authored-by: Hao Zhang <zhrt1446384557@gmail.com>

Fix bug produced by #339 Co-authored-by: Xing Guo admin@higuoxing.com

fix pipeline resource

# Problem Recording active tables and permission checking on segments rely on `monitored_dbid_cache`. But after mirror switching, the data in shared memory is lost, and the above behaviors will be blocked. # Solution Segments update their `monitored_dbid_cache` after receiving pull_active_tables_oid requests every naptime.

To achieve upgrading directly from 2.0 to 2.2, we should do the following things: - cherry-pick this PR in diskquota-2.0, diskquota-2.1 and diskquota-2.2. - set the shared_preload_libraries as `diskquota-2.2`: `gpconfig -c shared_preload_libraries -v 'diskquota-2.2'` - restart cluster: `gpstop -ar` - execute the following SQLs: ``` ALTER extension diskquota update to '2.2'; ```

Get the latest package version for release resource.

- Because of 97f1f9b, the sql file directory has been changed. The sql version check wouldn't work since it cannot find the sql files anymore. Change it to the correct ddl directory. - 'exec_program' is deprecated, use 'execute_process' instead. - 'git tag | sort' returns the latest version among all branches, but not the closest tag to the current commit. Use 'git describe --tags' instead. So the upgrade version check will work for the 2.1.x patch release.

After creating extension, the `diskquota.state` is always `unready` due to the change https://github.com/greenplum-db/gpdb/pull/15239. It makes the test timeout. We disable the job of gp7 in release/pr/merge pipeline until the problem is fixed.

- By #340, diskquota should be able to upgraded directly from any previous version. A script is added to test this. - Modify the cmakefile so before installing/packaing, only previous so files will copied. This would help us to make patch release for 2.0/2.1. --------- Co-authored-by: Zhang Hao <hzhang2@vmware.com>

Co-authored-by: Chen Mulong chenmulong@gmail.com

- bump version to 2.2.2 - Modify the check procedure of alter extension.

Fix from #346

git-subtree-dir: concourse/lib git-subtree-split: d51adf5d59e4e5caefb678b29c4335881040c7d2

…/lib'

Migrage changes https://github.com/pivotal/pgaudit/pull/13 here

Stolb27 · 2023-07-25T13:06:58Z

@RekGRpth, as far I understand, your patch 472eb06 still actual despite upstream modifications from 05da9d4 and 86ff586 because diskquota still loads shared libraries of intermediate versions during executing CREATE FUNCTION in the extension script. They solved the problem by ignoring initialization logic. But they expect that user preserve previous versions of the library.

…#20)" to reaaply above c2686c9 This reverts commit 3b06e37.

During diskquota worker's first run the initial set of active tables with their sizes is being loaded from diskquota.table_size table in order to warm up diskquota rejectmap and other shared memory objects. If an error occurs during this initialization process, the error will be ignored in PG_CATCH() block. Because of that local_active_table_stat_map will not be filled properly. And at the next loop iteration tables, that are not in acitive table list will be marked as irrelevant and to be deleted both from table_size_map and table_size table in flush_to_table_size function. In case when the inital set of active tables is huge (thousands of tables), this error ignorance could lead to the formation of a too long delete statement, which the SPI executor won't be able to process due to memory limits. And this case can lead to worker's segmentation fault or other errorneous behaviour of whole extension. This commit proposes the handling of the initialization errors, which occur during worker's first run. In the DiskquotaDBEntry structure the bool variable "corrupted" is added in order to indicate, that the worker wasn't able to initialize itself on given database. And DiskquotaDBEntry also is now passed to refresh_disk_quota_model function from worker main loop, because one need to change the state of dbEntry. The state is changed when the refresh_disk_quota_usage function catches an error, which occured during the initialization step, in PG_CATCH() block. And after the error is catched, the "corrupted" flag is set in given dbEntry, and then the error is rethrown. This leads to worker process termination. The launcher will not be able to start it again, because added flag is set in the database structure, and this flag is being checked inside the disk_quota_launcher_main function. The flag can be reseted by calling resetBackgroundWorkerCorruption function, which is currently called in SIGHUP handler. Cherry-picked-from: 3b06e37 to reapply above c2686c9

src/diskquota.h

RekGRpth · 2023-07-26T03:26:59Z

@red1452, these commits 05da9d4 and 86ff586 seem to solve the problem #21 (comment).

red1452 · 2023-07-26T08:18:48Z

src/diskquota.c

+	else
+		match = false;
+	pfree(query);
+	return match;


This function is incorrect if it is necessary that ActivePortal->sourceText must contain word update. This word is not checked at this function.

This word is not checked at this function.

Why?

After save pos at line 192 there is check pos is not null at line 195. This is check that query contains diskquota. If it is true, match is not change to false. Next saving pointer of update to pos does not affect to variable match. And true will be returned even if strstr(pos, "update"); returns NULL.

Can you explain step by step?

182: query = "alter extension diskquota update" 184: match = true 186: pos = "alter extension diskquota update" 188: pos = "extension diskquota update" 192: pos = "diskquota update" 196: pos = "update" 200: match = true

If you type another word instead of update you will get match = true but expect match = false.

Exactly! If last word is not update then

196: pos = NULL

They missed another check

if (pos) pos = strstr(pos, "update"); else match = false; if (!pos) match = false;

existing upgrade and alter tests are useless for ADB: - previous releases incompatible with current due PG-module version check - our upgrade process exclude existing of previous versions during upgrade

Stolb27 · 2023-07-27T13:03:09Z

Currently, diskquota 2.2.2 is affected by greenplum-db#360. Delayed.

Stolb27 · 2023-07-27T13:10:42Z

src/diskquota.c

+{
+	if (ActivePortal == NULL) return false;
+	/* QD: When the sourceTag is T_AlterExtensionStmt, then return true */
+	if (ActivePortal->sourceTag == T_AlterExtensionStmt) return true;


But we don't return false on QD otherwise. As a result, we anyway validate sourceText that may contain any garbage.

Stolb27 · 2023-12-05T10:24:28Z

Closed in favor of #26

higuoxing and others added 18 commits May 25, 2023 17:30

Fix bug: diskquota.status() (#344)

1111f71

Fix bug produced by #339 Co-authored-by: Xing Guo admin@higuoxing.com

fix pipeline (#345)

58bb9e2

fix pipeline resource

Reduce the number of the log in bgworker. (#346)

c2686c9

Fix pipeline. (#349)

c81e696

Get the latest package version for release resource.

Remove gp7 from pipeline (#350)

974876d

After creating extension, the `diskquota.state` is always `unready` due to the change https://github.com/greenplum-db/gpdb/pull/15239. It makes the test timeout. We disable the job of gp7 in release/pr/merge pipeline until the problem is fixed.

Add a sleep in alter_test.sh (#351)

2aefa80

Co-authored-by: Chen Mulong chenmulong@gmail.com

Update to 2.2.2 (#352)

86ff586

- bump version to 2.2.2 - Modify the check procedure of alter extension.

Reduce the remain logs in bgworker. (#354)

f1ca0c5

Fix from #346

Fix bug: bgworkers only print log once. (#356)

2c086fc

Update resources by using gp-extensions-ci subtree

22c35b6

Squashed 'concourse/lib/' content from commit d51adf5

dbd044f

git-subtree-dir: concourse/lib git-subtree-split: d51adf5d59e4e5caefb678b29c4335881040c7d2

Merge commit 'dbd044f46d7672331e19cbd6b95a6f8df67511af' as 'concourse…

6327f5e

…/lib'

Update resources by using gp-extensions-ci subtree (#357)

bb840f5

Migrage changes https://github.com/pivotal/pgaudit/pull/13 here

deart2k requested a review from Stolb27 July 25, 2023 08:05

Stolb27 and others added 3 commits July 25, 2023 15:10

Revert "ADBDEV-3685 Error handling for disqkuota worker startup stage (…

d6cae3c

…#20)" to reaaply above c2686c9 This reverts commit 3b06e37.

Merge branch '2.2.2-conflicts' into 2.2.2-sync

624e3e7

Stolb27 requested review from RekGRpth and bimboterminator1 July 25, 2023 13:29

bimboterminator1 reviewed Jul 25, 2023

View reviewed changes

src/diskquota.h Outdated Show resolved Hide resolved

RekGRpth previously approved these changes Jul 26, 2023

View reviewed changes

red1452 reviewed Jul 26, 2023

View reviewed changes

mute alter extension tests

16fecbb

existing upgrade and alter tests are useless for ADB: - previous releases incompatible with current due PG-module version check - our upgrade process exclude existing of previous versions during upgrade

Stolb27 dismissed RekGRpth’s stale review via 16fecbb July 26, 2023 14:50

Stolb27 marked this pull request as draft July 27, 2023 13:03

Stolb27 reviewed Jul 27, 2023

View reviewed changes

Stolb27 closed this Dec 5, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ADBDEV-4015 2.2.2 sync#23

ADBDEV-4015 2.2.2 sync#23
deart2k wants to merge 22 commits intogpdbfrom
2.2.2-sync

deart2k commented Jul 25, 2023

Uh oh!

Stolb27 commented Jul 25, 2023

Uh oh!

Uh oh!

RekGRpth commented Jul 26, 2023

Uh oh!

red1452 Jul 26, 2023

Uh oh!

RekGRpth Jul 26, 2023

Uh oh!

red1452 Jul 26, 2023

Uh oh!

RekGRpth Jul 26, 2023

Uh oh!

red1452 Jul 26, 2023

Uh oh!

RekGRpth Jul 26, 2023

Uh oh!

Stolb27 commented Jul 27, 2023

Uh oh!

Stolb27 Jul 27, 2023

Uh oh!

Stolb27 commented Dec 5, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

Conversation

deart2k commented Jul 25, 2023

Uh oh!

Stolb27 commented Jul 25, 2023

Uh oh!

Uh oh!

RekGRpth commented Jul 26, 2023

Uh oh!

red1452 Jul 26, 2023

Choose a reason for hiding this comment

Uh oh!

RekGRpth Jul 26, 2023

Choose a reason for hiding this comment

Uh oh!

red1452 Jul 26, 2023

Choose a reason for hiding this comment

Uh oh!

RekGRpth Jul 26, 2023

Choose a reason for hiding this comment

Uh oh!

red1452 Jul 26, 2023

Choose a reason for hiding this comment

Uh oh!

RekGRpth Jul 26, 2023

Choose a reason for hiding this comment

Uh oh!

Stolb27 commented Jul 27, 2023

Uh oh!

Stolb27 Jul 27, 2023

Choose a reason for hiding this comment

Uh oh!

Stolb27 commented Dec 5, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants