2.31.0.0-b225
tagged this
02 Jul 20:48
Summary:
This diff enables the explicit LOAD command into YugabyteDB, from [[ https://www.postgresql.org/docs/15/sql-load.html | postgres ]]. LOAD is a command that loads a shared library file (.so file) into a backend’s address space. It is superuser-trusted, so only users with permission can load the shared library and “non-superusers can only apply LOAD to library files located in $libdir/plugins/In” in a white-list fashion. In YugabyteDB, a library must be loaded safely per-backend, so that the files only load into the calling session’s process. It should not be cluster-global to prevent conflicts between shared data.
Hooks that can be loaded into the backend are exclusively compiled shared library files (.so) containing C code (unlike extensions which include .sql, .control, and c library file). Example of LOAD is with the `auto_explain` library/hook. auto_explain is a module that logs execution plans of queries automatically (like EXPLAIN to a log file). `auto_explain` works purely by installing executor hooks in its `_PG_init()`, which only runs when the library is loaded into the backend. Because such a hook module exposes no SQL-callable objects, there is nothing for the server to lazily load on first use the way an extension is loaded when one of its functions is first called. So the hooks cannot be lazily loaded on a user command, and `LOAD` (or `shared_preload_libraries`) is the only way to install them into a running session.
The LOAD statement was already implemented, so `parser_ybc_not_supported` needed to be removed to enable the command. A connection manager use case (to mark the connection as sticky) was also added, to ensure persistence of the library loaded.
The current implementation of LOAD does not require distributed/persistent handling; it is backend-local and session-scoped. Loaded files are stored to a local linked list (file_list in src/postgres/src/backend/utils/fmgr/dfmgr.c), and never in shared memory or catalog (parallel workers copy it over, and do not send libraries to other nodes). The state lives for the lifetime of the backend session, and the shared libraries are never propagated to backends on other nodes.
However, when Connection Manager is enabled, the file/extension loaded only persists in the physical backend that ran it, so later statements may be routed to other physical backends that don’t have the LOAD state, producing incorrect outputs. This diff creates a safeguard: mark the connection as sticky when Connection Manager is enabled, when LOAD statement is executed
For the following reasons, LOAD can be safely enabled:
# It introduces no persistent state to manage
# YugabyteDB is already using the `internal_load_library()` function
# The permission model is intact, meaning it adds capability without adding distributed state or new privilege
Github issue: https://github.com/yugabyte/yugabyte-db/issues/9438
------------------------------------
Before:
```
ERROR: LOAD not supported yet
LINE 1: load 'auto_explain'
```
After:
```
yugabyte=# LOAD 'auto_explain';
LOAD
yugabyte=# SET auto_explain.log_analyze=true;
SET
yugabyte=# SET auto_explain.log_min_duration = 0;
SET
yugabyte=# SET client_min_messages = log;
SET
yugabyte=# SELECT count(*)
yugabyte-# FROM pg_class, pg_index
yugabyte-# WHERE oid = indrelid AND indisunique;
LOG: duration: 63.194 ms plan:
Query Text: SELECT count(*)
FROM pg_class, pg_index
WHERE oid = indrelid AND indisunique;
Aggregate (cost=26.38..26.39 rows=1 width=8) (actual time=62.294..62.295 rows=1 loops=1)
-> Nested Loop (cost=0.01..26.06 rows=129 width=0) (actual time=7.580..62.247 rows=119 loops=1)
-> Seq Scan on pg_index (cost=0.00..12.90 rows=129 width=4) (actual time=6.525..6.645 rows=119 loops=1)
Storage Filter: indisunique
-> Memoize (cost=0.01..0.15 rows=1 width=4) (actual time=0.465..0.465 rows=1 loops=119)
Cache Key: pg_index.indrelid
Cache Mode: logical
Hits: 48 Misses: 71 Evictions: 0 Overflows: 0 Memory Usage: 8kB
-> Index Scan using pg_class_oid_index on pg_class (cost=0.00..0.14 rows=1 width=4) (actual time=0.747..0.747 rows=1 loops=71)
Index Cond: (oid = pg_index.indrelid)
```
Test Plan:
There are some tests that used the shared_preload_libraries instead of the LOAD statement to test LOAD functionality.
The following tests can now be changed to implement LOAD:
- `src/postgres/src/pl/plperl/sql/plperl_init.sql`
- `src/postgres/src/pl/plperl/sql/plperlu.sql`
- `src/postgres/src/test/modules/test_oat_hooks/sql/test_oat_hooks.sql`
- `src/postgres/contrib/sepgsql/sql/misc.sql`
- `src/postgres/src/test/modules/test_rls_hooks/sql/test_rls_hooks.sql` (this PR ports the LOAD version)
- `src/postgres/src/test/regress/sql/guc.sql`
Run tests:
- `./yb_build.sh release --java-test 'org.yb.pgsql.TestPgRegressModulesTestRlsHooks#schedule'`
- `./yb_build.sh fastdebug --java-test 'org.yb.pgsql.TestPgAutoExplain'`
- `./yb_build.sh fastdebug --java-test 'org.yb.pgsql.TestPgRegressContribPasswordCheck' --enable-ysql-conn-mgr-test`
- `./yb_build.sh fastdebug --java-test 'org.yb.ysqlconnmgr.TestConnectionManagerLoadPasswordcheck'`
- `./yb_build.sh fastdebug --java-test 'org.yb.pgsql.TestPgRegressThirdPartyExtensionsPgHintPlan#schedule'`
- `./yb_build.sh fastdebug --java-test 'org.yb.pgsql.TestPgRegressPgMisc#testPgRegressPgMisc'`
- `./yb_build.sh fastdebug --java-test 'org.yb.pgsql.TestPgRegressPlpgsql#testPgRegressPlpgsql'`
```
# test LOAD 'auto_explain' in postgres
SET client_min_messages = log;
LOAD 'auto_explain';
SET auto_explain.log_min_duration = 0;
SET auto_explain.log_analyze = true;
SELECT count(*)
FROM pg_class, pg_index
WHERE oid = indrelid AND indisunique;
```
- Written test TestConnectionManagerLoadPasswordcheck for verifying that connection stays sticky when connection manager is enabled in a LOAD session
- Created regression tests TestPgLoadAutoExplain for `auto_explain` extension
- Updated existing tests outputs that use LOAD
Reviewers: kramanathan, mkumar
Reviewed By: kramanathan
Subscribers: mkumar, jason, yql
Tags: #jenkins-ready
Differential Revision: https://phorge.dev.yugabyte.com/D54054