Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[YSQL] Add flag controlling relcache file usage #19226

Closed
1 task done
kai-franz opened this issue Sep 21, 2023 · 0 comments
Closed
1 task done

[YSQL] Add flag controlling relcache file usage #19226

kai-franz opened this issue Sep 21, 2023 · 0 comments
Assignees
@kai-franz
Labels
area/ysql Yugabyte SQL (YSQL) kind/enhancement This is an enhancement of an existing feature priority/medium Medium priority issue

Comments

@kai-franz
Copy link
Contributor

kai-franz commented Sep 21, 2023 •
edited by jira bot

Jira Link: DB-8018

Description

Currently, the first Postgres backend that starts will preload its catalog cache in order to build the relcache init file. Subsequent connections do not preload their catalog cache, instead relying on the previously built relcache init file and on-demand RPC requests to master during query execution.

This behavior is undesirable for two reasons:

  1. Unpredictable memory usage: Postgres backend memory usage becomes dependent on whether the catalog caches are refreshed. By default, subsequent backend processes have low memory usage because they don't preload their catalog caches. However, if a DDL is executed after these connections are established, they will need to refresh their catalog caches, using more memory.
  2. Higher overhead on master: Catalog cache lookups will be replaced with on-demand RPCs to master during query execution.

To remedy this, we would like to add a flag that controls whether the relcache file is used, allowing customers to choose between (a) predictable memory usage/low master overhead and (b) lower memory usage during DML-only workloads.

Warning: Please confirm that this issue does not contain any sensitive information

  • I confirm this issue does not contain any sensitive information.
@kai-franz kai-franz added area/ysql Yugabyte SQL (YSQL) status/awaiting-triage Issue awaiting triage labels Sep 21, 2023
@kai-franz kai-franz self-assigned this Sep 21, 2023
@yugabyte-ci yugabyte-ci added kind/enhancement This is an enhancement of an existing feature priority/medium Medium priority issue labels Sep 21, 2023
@yugabyte-ci yugabyte-ci removed the status/awaiting-triage Issue awaiting triage label Oct 16, 2023
kai-franz added a commit that referenced this issue Oct 18, 2023
@kai-franz
[#19226] YSQL: Add ysql_use_relcache flag, don't preload additional t…
ab273df 
…ables when ysql_enable_read_request_caching=true

Summary:
Changes the

1. D22407 introduced behavior that unconditionally preloads certain system tables when `ysql_enable_read_request_caching` is enabled, which can significantly increase PG backend process memory usage. This diff reverts those changes, reducing memory usage.
2. Changes how the relcache init file is used and introduces a new flag, `ysql_use_relcache_file`, which defaults to true.

The use of the relcache init file has been modified to the following:
- If `ysql_catalog_preload_additional_tables` is set to true or `ysql_catalog_preload_additional_table_list` is nonempty, we automatically skip both reading from and creating the relcache file.
- Otherwise, we read from/create the relcache file only if `ysql_use_relcache_file` is set to true, which is its default setting.

Test Plan: - `pg_catalog_perf-test.cc`: Tests setting `ysql_use_relcache_file = false`.

Reviewers: fizaa, myang, dmitry

Reviewed By: myang, dmitry

Subscribers: bogdan, jason, tverona, kfranz, yql

Differential Revision: https://phorge.dev.yugabyte.com/D28071
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kai-franz kai-franz

Labels
area/ysql Yugabyte SQL (YSQL) kind/enhancement This is an enhancement of an existing feature priority/medium Medium priority issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@kai-franz @yugabyte-ci