fix(db/declarative): fix TTL not working in DB-less and Hybrid mode #11464

chobits · 2023-08-28T03:56:41Z

Summary

Previously, in DB-less and Hybrid mode, the ttl/updated_at fields were not copied from the original entities to the flattened entities. As a result, the entities were loaded without the TTL field.
Additionally, for loading the TTL field, the "off" DB strategy (lmdb) did not support automatic calculation of the remaining expiration time.

Checklist

The Pull Request has tests
There's an entry in the CHANGELOG
There is a user-facing docs PR against https://github.com/Kong/docs.konghq.com - PUT DOCS PR HERE

Full changelog

copy the ttl field from original entities to flatten entities
transform the absolute ttl value to relative within the new function each_for_export(), then declarative config exports its configuration using for ... in each_for_export instead of for ... in each()
add DAO:each_for_export() based on DAO:each() to export the entity with the absolute ttl
- add the options.export field to enable this feature in pg:page()
- add extra pg SQL statements "page_XXX_for_export{_global}"
the off strategy:
- turn the absolute ttl to relative for the upper user (cred.ttl) in off:select()/:page()
- does not return the expired entity(cred) to the upper user, which makes it identical with the PG strategy.

Issue reference

Fix FTI-4512

kong/db/schema/others/declarative_config.lua

kong/db/strategies/off/init.lua

chronolaw · 2023-08-28T05:44:39Z

We also need a change log entry.

kong/db/schema/others/declarative_config.lua

chobits · 2023-08-31T02:20:18Z

kong/db/declarative/export.lua

@@ -142,7 +142,7 @@ local function export_from_db_impl(emitter, skip_ws, skip_disabled_entities, exp
    if db[name].pagination then
      page_size = db[name].pagination.max_page_size
    end
-    for row, err in db[name]:each(page_size, GLOBAL_QUERY_OPTS) do
+    for row, err in db[name]:each_for_export(page_size, GLOBAL_QUERY_OPTS) do


As discussed with bungle, we have used :each_for_export here to export the entity with the absolute ttl while keeping original :each unchanged. :each will continue to give the relative.

CHANGELOG.md

https://github.com/Kong/kong/blob/master/kong/db/strategies/postgres/init.lua -> add each_for_export https://github.com/Kong/kong/blob/master/kong/db/declarative/export.lua#L145 -> use :each_for_export https://github.com/Kong/kong/blob/master/kong/db/declarative/import.lua#L167 -> ensure that ttl is properly stored in lmdb (absolute value) https://github.com/Kong/kong/blob/master/kong/db/strategies/off/init.lua -> make sure that TTL is respected (expired items are not returned), and turn absolute ttl to relative, and ensure that it is returned (the cred.ttl)

We have already introduced CHANGELOG/unreleased/kong/11464.yaml.

* sleep before attempting unnecessary requests * decrease the ttl to expedite the case's execution

bungle

Looks good!

…11464) * fix(declarative): fix TTL not working in DB-less and Hybrid mode * Previously, in DB-less and Hybrid mode, the ttl/updated_at fields were not copied from the original entities to the flattened entities. As a result, the entities were loaded without the TTL field. * Additionally, for loading the TTL field, the "off" DB strategy (lmdb) did not properly filter expired items, nor returned right TTL value for DAO. FTI-4512 * fix coding style * fix coding style: improved function name * added test case: hybrid mode for key-auth * fix test case warnings * fixed test case consumer domain * export ttl as absolute value * delete unused defination * move ttl-fixing logic into row_to_entity() * still use pg to caculate relative value * clean code * add changelog entry * fixed test cases * fixed test cases warning * fixed test failure * fix test case issue: ttl expiration * fix test case: unsed local variable * add an entry in CHANGELOG.md * fix changelog scope * remove release-related information in CHANGELOG.md * fix test case: sleep before attempting unnecessary requests * sleep before attempting unnecessary requests * decrease the ttl to expedite the case's execution * fix CHANGELOG typo * fix the tense problem of changelog entry * add export options for "page_*_for_export" sql statement * fix warning: setting non-standard global variable * fix error reporting: options is nil * fix an issue where the off strategy returned the expired entity * run ttl processing before schema:process_auto_fields()

pull-request-size bot added the size/M label Aug 28, 2023

This comment was marked as outdated.

Sign in to view

chobits requested review from bungle and chronolaw August 28, 2023 03:56

github-actions bot added core/db schema-change-noteworthy labels Aug 28, 2023

This comment was marked as resolved.

Sign in to view

chobits requested review from StarlightIbuki and removed request for bungle August 28, 2023 03:57

chobits force-pushed the fix_ttl branch 2 times, most recently from 2ca8583 to 8e7daba Compare August 28, 2023 05:05

This comment was marked as resolved.

Sign in to view

chronolaw reviewed Aug 28, 2023

View reviewed changes

kong/db/schema/others/declarative_config.lua Outdated Show resolved Hide resolved

kong/db/schema/others/declarative_config.lua Outdated Show resolved Hide resolved

kong/db/strategies/off/init.lua Outdated Show resolved Hide resolved

chronolaw changed the title ~~fix(declarative): fix TTL not working in DB-less and Hybrid mode~~ fix(db/declarative): fix TTL not working in DB-less and Hybrid mode Aug 28, 2023

pull-request-size bot added size/S and removed size/M labels Aug 28, 2023

chobits requested a review from bungle August 28, 2023 06:24

pull-request-size bot added size/L and removed size/S labels Aug 29, 2023

chronolaw reviewed Aug 29, 2023

View reviewed changes

kong/db/schema/others/declarative_config.lua Outdated Show resolved Hide resolved

chobits force-pushed the fix_ttl branch from 923316b to 4ec3116 Compare August 29, 2023 06:26

AndyZhang0707 marked this pull request as draft August 29, 2023 08:33

chobits force-pushed the fix_ttl branch from 4ec3116 to 55e97dc Compare August 29, 2023 16:08

This comment was marked as outdated.

Sign in to view

github-actions bot added the changelog label Aug 30, 2023

chobits marked this pull request as ready for review August 31, 2023 02:12

chobits commented Aug 31, 2023

View reviewed changes

chobits force-pushed the fix_ttl branch from 4a752aa to f24b0b7 Compare August 31, 2023 02:38

chronolaw reviewed Aug 31, 2023

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

chobits added 22 commits September 7, 2023 12:31

export ttl as absolute value

42a7b2f

delete unused defination

ca979f2

move ttl-fixing logic into row_to_entity()

5799a81

clean code

2d90e5d

add changelog entry

c57d259

fixed test cases

f122f06

fixed test cases warning

93cc9f9

fixed test failure

745c659

fix test case issue: ttl expiration

e3c3ea8

fix test case: unsed local variable

cc0c81a

add an entry in CHANGELOG.md

0125ab6

fix changelog scope

7e73638

remove release-related information in CHANGELOG.md

8cff12c

We have already introduced CHANGELOG/unreleased/kong/11464.yaml.

fix test case: sleep before attempting unnecessary requests

40c0c22

* sleep before attempting unnecessary requests * decrease the ttl to expedite the case's execution

fix CHANGELOG typo

e9d22c8

fix the tense problem of changelog entry

2525c56

add export options for "page_*_for_export" sql statement

ab19f53

fix warning: setting non-standard global variable

600498a

fix error reporting: options is nil

810306e

fix an issue where the off strategy returned the expired entity

f971ef7

run ttl processing before schema:process_auto_fields()

91f1294

chobits force-pushed the fix_ttl branch from 9dff899 to 91f1294 Compare September 7, 2023 04:31

bungle approved these changes Sep 7, 2023

View reviewed changes

bungle merged commit 48dc2ef into Kong:master Sep 7, 2023
24 checks passed

chobits mentioned this pull request Sep 27, 2023

Kong DP node killed with SIGQUIT during startup with large configuration #11651

Closed

1 task

windmgc mentioned this pull request Jan 24, 2024

[cherry release/3.4.x] cherry-picking important fixes to 34x #12403

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(db/declarative): fix TTL not working in DB-less and Hybrid mode #11464

fix(db/declarative): fix TTL not working in DB-less and Hybrid mode #11464

chobits commented Aug 28, 2023 •

edited

This comment was marked as outdated.

This comment was marked as resolved.

This comment was marked as resolved.

chronolaw commented Aug 28, 2023

This comment was marked as outdated.

chobits Aug 31, 2023 •

edited

bungle left a comment

fix(db/declarative): fix TTL not working in DB-less and Hybrid mode #11464

fix(db/declarative): fix TTL not working in DB-less and Hybrid mode #11464

Conversation

chobits commented Aug 28, 2023 • edited

Summary

Checklist

Full changelog

Issue reference

This comment was marked as outdated.

This comment was marked as resolved.

This comment was marked as resolved.

chronolaw commented Aug 28, 2023

This comment was marked as outdated.

chobits Aug 31, 2023 • edited

Choose a reason for hiding this comment

bungle left a comment

Choose a reason for hiding this comment

chobits commented Aug 28, 2023 •

edited

chobits Aug 31, 2023 •

edited