feat(server): Allow configuration of hashtag extraction #2890

chakaz · 2024-04-11T19:33:51Z

Before this PR: Hashtags, if enabled, meant the text between { and } in the key (if exists and non-empty).

After this PR:

Hashtags can still be enabled / disabled
Hashtag open / close char can be specified (and can be the same), like : popular with BullMQ
Hashtag can include N closing tags to skip, like {a}b}c}d with 2 will return a}b}c.
Hashtags can be limited to only be applied on keys with a specified prefix

This will allow some existing systems to migrate without code changes in client code.

Before this PR: Hashtags, if enabled, meant the text between `{` and `}` in the key (if exists and non-empty). After this PR: * Hashtags can _still_ be enabled / disabled * Hashtag open / close char can be specified (and can be the same), like `:` popular with BullMQ * Hashtag can include `N` closing tags to skip, like `{a}b}c}d` with `2` will return `a}b}c`. This will allow some existing systems to migrate without code changes in client code.

romange · 2024-04-14T07:16:35Z

src/server/common.cc

+ABSL_FLAG(std::string, hashtag_open, "{", "Opening hashtag character. Must be a single char.");
+ABSL_FLAG(std::string, hashtag_close, "}", "Closing hashtag character. Must be a single char.");
+
+ABSL_FLAG(std::string, hashtag_prefix, "",


queue_prefix

locktag_prefix

romange · 2024-04-14T07:19:57Z

src/server/common.cc

@@ -29,33 +29,57 @@ extern "C" {
 ABSL_FLAG(bool, lock_on_hashtags, false,
          "When true, locks are done in the {hashtag} level instead of key level.");

+// We would have used `char` instead of `string` for the following 2 flags, but that's impossible.
+ABSL_FLAG(std::string, hashtag_open, "{", "Opening hashtag character. Must be a single char.");


locktag_delimiter

src/server/common.cc

romange · 2024-04-14T10:40:45Z

src/server/common.cc

+
+ABSL_FLAG(unsigned, locktag_skip_n_end_delimiters, 0,
+          "How many closing hashtags should we skip. 0 for no skipping. For example, when set to "
+          "2, the hashtag for '{a}b}c}d}e' will be 'a}b}c'.");


I suggest changing the example to ':' delimiters - it's more clear and relevant, imho.

romange · 2024-04-14T10:49:13Z

src/server/common.h

 struct KeyLockArgs {
-  static bool IsLockHashTagEnabled();
+  static LockTagOptions GetLockTagOptions();


I suggest keeping IsLockHashTagEnabled and move LockTagOptions to cc

But we need to expose these options for cluster_config, where hash tag extraction is done.
We can move it back, and maybe rename the KeyTag() function while on it, but it's kinda beyond the scope here I think.

…nfly ( v1.16.1 → v1.17.0 ) (#3473) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [docker.dragonflydb.io/dragonflydb/dragonfly](https://togithub.com/dragonflydb/dragonfly) | minor | `v1.16.1` -> `v1.17.0` | --- ### Release Notes <details> <summary>dragonflydb/dragonfly (docker.dragonflydb.io/dragonflydb/dragonfly)</summary> ### [`v1.17.0`](https://togithub.com/dragonflydb/dragonfly/releases/tag/v1.17.0) [Compare Source](https://togithub.com/dragonflydb/dragonfly/compare/v1.16.1...v1.17.0) ##### Dragonfly v1.17.0 Some prominent changes include: - Improved performance for MGET operations ([#2453](https://togithub.com/dragonflydb/dragonfly/issues/2453)) - Fix argument parsing in json.objkeys ([#2872](https://togithub.com/dragonflydb/dragonfly/issues/2872)) - Fix ipv6 support for replication ([#2889](https://togithub.com/dragonflydb/dragonfly/issues/2889)) - Support serialisation of bloom filters - saving to and loading from snapshots ([#2846](https://togithub.com/dragonflydb/dragonfly/issues/2846)) - Support of HLL PFADD ([#2761](https://togithub.com/dragonflydb/dragonfly/issues/2761)) - Support bullmq workloads that do not have `{}` hashtags in their queue names ([#2890](https://togithub.com/dragonflydb/dragonfly/issues/2890)) ##### What's Changed - fix: [#2745](https://togithub.com/dragonflydb/dragonfly/issues/2745) don't start migration process again after apply the same the same config is applied by [@BorysTheDev](https://togithub.com/BorysTheDev) in [dragonflydb/dragonfly#2822 - feat(transaction): Idempotent callbacks (immediate runs) by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#2453 - refactor(cluster): replace sync_id with node_id for slot migration [#2835](https://togithub.com/dragonflydb/dragonfly/issues/2835) by [@BorysTheDev](https://togithub.com/BorysTheDev) in [dragonflydb/dragonfly#2838 - feat(tiering): Simple OpManager by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#2781 - chore: implement path mutation for JsonFlat by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2805 - feat(cluster): add migration removing by config [#2835](https://togithub.com/dragonflydb/dragonfly/issues/2835) by [@BorysTheDev](https://togithub.com/BorysTheDev) in [dragonflydb/dragonfly#2844 - chore: expose direct API on Bloom objects by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2845 - chore: generalize CompactObject::AllocateMR by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2847 - feat(tiering): Simplest small bins by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#2810 - refactor: clean cluster slot migration code by [@BorysTheDev](https://togithub.com/BorysTheDev) in [dragonflydb/dragonfly#2848 - fix(tests): Fix numsub test by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#2852 - fix: healthcheck for docker containers by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2853 - fix: possible crash in tls code by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2854 - fix(server): Do not block admin-port commands by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#2842 - fix(pytest): make pytests fail if server crash on shutdown by [@adiholden](https://togithub.com/adiholden) in [dragonflydb/dragonfly#2827 - feat(server): add prints on takeover timeout by [@adiholden](https://togithub.com/adiholden) in [dragonflydb/dragonfly#2856 - fix(pytest): dont check process return code on kill by [@adiholden](https://togithub.com/adiholden) in [dragonflydb/dragonfly#2862 - fix: authorize the http connection to call commands by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2863 - feat(cluster): Send number of keys for incoming and outgoing migrations. by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#2858 - feat(tiering): TieredStorageV2 by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#2849 - bug(server): set connection flags block/pause flag on all blocking commands by [@adiholden](https://togithub.com/adiholden) in [dragonflydb/dragonfly#2816 - chore: serialize SBF by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2846 - fix: test_replicaof_reject_on_load crash on stop by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2818 - feat(dbslice): Add self-laundering iterator in `DbSlice` by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#2815 - chore: License update by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2767 - fix(acl): incompatibilities with acl load by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2867 - fix(json): make path optional in json.objkeys by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2872 - fix: return wrong type errors for SET...GET command by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2874 - fix(redis replication): remove partial sync flow ,not supported yet by [@adiholden](https://togithub.com/adiholden) in [dragonflydb/dragonfly#2865 - chore: limit traffic logger only to the main interface by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2877 - chore: relax repltakeover constraints to only exclude write commands by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2873 - chore(replayer): Roll back to go1.18 by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#2881 - fix: brpoplpush single shard to wake up blocked transactions by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2875 - chore: LockTable tracks fingerprints of keys by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2839 - chore: reject TLS handshake when our listener is plain TCP by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2882 - Add support for Sparse HLL PFADD by [@azuredream](https://togithub.com/azuredream) in [dragonflydb/dragonfly#2761 - feat server: bring visibility to script errors by [@adiholden](https://togithub.com/adiholden) in [dragonflydb/dragonfly#2879 - chore: clean up REPLTAKEOVER flow by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2887 - chore(tiering): Move files and move kb literal to common by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#2868 - chore(interpreter): Support object replies by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#2885 - fix(ci/helm): Stick to v0.73.0 version of prom operator by [@Pothulapati](https://togithub.com/Pothulapati) in [dragonflydb/dragonfly#2893 - fix(acl): authentication with UDS socket by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2895 - feat(cluster): add repeated ACK if an error is happened by [@BorysTheDev](https://togithub.com/BorysTheDev) in [dragonflydb/dragonfly#2892 - chore(blocking): Remove faulty DCHECK by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#2898 - chore: add a clear link on how to build dragonfly from source by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2884 - feat(server): Allow configuration of hashtag extraction by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#2890 - fix: fix build under macos by [@BorysTheDev](https://togithub.com/BorysTheDev) in [dragonflydb/dragonfly#2901 - fix(cluster_replication): replicate redis cluster node bug fix by [@adiholden](https://togithub.com/adiholden) in [dragonflydb/dragonfly#2876 - fix(acl): skip http and add check on connection traversals by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2883 - fix(zset): Better memory consumption calculation by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#2900 - fix: fix ld for num converting by [@BorysTheDev](https://togithub.com/BorysTheDev) in [dragonflydb/dragonfly#2902 - chore: add help string for memory_fiberstack_vms_bytes by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2903 - fix(sanitizers): false positive fail on multi_test::Eval by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2896 - chore: pull helio and add ipv6 replication test by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#2889 - chore: add ipv6 support for native linux release by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2908 ##### Huge thanks to all the contributors! ❤️ **Full Changelog**: dragonflydb/dragonfly@v1.16.0...v1.17.0 </details>  Co-authored-by: repo-jeeves[bot] <106431701+repo-jeeves[bot]@users.noreply.github.com>

chakaz added 3 commits April 11, 2024 22:33

add hashtag prefix setting

4623a22

Merge branch 'main' into hashtag-options

654fca1

chakaz requested a review from romange April 14, 2024 06:06

romange reviewed Apr 14, 2024

View reviewed changes

src/server/common.cc Outdated Show resolved Hide resolved

chakaz added 3 commits April 14, 2024 11:32

Merge branch 'main' into hashtag-options

6d288b6

renames etc

7e0d26f

more renames

2baf217

chakaz requested a review from romange April 14, 2024 10:13

romange reviewed Apr 14, 2024

View reviewed changes

romange previously approved these changes Apr 14, 2024

View reviewed changes

docs

dbec7a4

chakaz dismissed romange’s stale review via dbec7a4 April 14, 2024 10:52

chakaz requested a review from romange April 14, 2024 10:54

hashtag

6d6c5e0

romange approved these changes Apr 14, 2024

View reviewed changes

romange merged commit 256e07f into main Apr 14, 2024
10 checks passed

romange deleted the hashtag-options branch April 14, 2024 12:09

Pothulapati mentioned this pull request Apr 19, 2024

Namespace support in prometheus rule #2931

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(server): Allow configuration of hashtag extraction #2890

feat(server): Allow configuration of hashtag extraction #2890

chakaz commented Apr 11, 2024 •

edited

Loading

romange Apr 14, 2024

romange Apr 14, 2024

romange Apr 14, 2024

romange Apr 14, 2024

romange Apr 14, 2024

chakaz Apr 14, 2024

feat(server): Allow configuration of hashtag extraction #2890

feat(server): Allow configuration of hashtag extraction #2890

Conversation

chakaz commented Apr 11, 2024 • edited Loading

romange Apr 14, 2024

Choose a reason for hiding this comment

romange Apr 14, 2024

Choose a reason for hiding this comment

romange Apr 14, 2024

Choose a reason for hiding this comment

romange Apr 14, 2024

Choose a reason for hiding this comment

romange Apr 14, 2024

Choose a reason for hiding this comment

chakaz Apr 14, 2024

Choose a reason for hiding this comment

chakaz commented Apr 11, 2024 •

edited

Loading