Refresh COMMAND, add topics: command-args, key-specs #1697

guybe7 · 2021-12-01T15:11:22Z

Add new COMMAND command features implemented in redis/redis#9656 and it's predecessor PRs (key-specs, and sub-commands)
Plus some other cleanup and improvements.

TODO:

Document COMMAND LIST
Write the key-specs topic
Write the command hints
Document key-spec-index (new field for command argument)
update the list of command flags

commands/command.md

commands/command-list.md

commands.json

commands/command-list.md

commands/command.md

topics/command-info.md

Co-authored-by: Itamar Haber <itamar@redis.com>

oranagra

please run this though a spell checker.

topics/key-specs.md

Delete the hardcoded command table and replace it with an auto-generated table, based on a JSON file that describes the commands (each command must have a JSON file). These JSON files are the SSOT of everything there is to know about Redis commands, and it is reflected fully in COMMAND INFO. These JSON files are used to generate commands.c (using a python script), which is then committed to the repo and compiled. The purpose is: * Clients and proxies will be able to get much more info from redis, instead of relying on hard coded logic. * drop the dependency between Redis-user and the commands.json in redis-doc. * delete help.h and have redis-cli learn everything it needs to know just by issuing COMMAND (will be done in a separate PR) * redis.io should stop using commands.json and learn everything from Redis (ultimately one of the release artifacts should be a large JSON, containing all the information about all of the commands, which will be generated from COMMAND's reply) * the byproduct of this is: * module commands will be able to provide that info and possibly be more of a first-class citizens * in theory, one may be able to generate a redis client library for a strictly typed language, by using this info. ### Interface changes #### COMMAND INFO's reply change (and arg-less COMMAND) Before this commit the reply at index 7 contained the key-specs list and reply at index 8 contained the sub-commands list (Both unreleased). Now, reply at index 7 is a map of: - summary - short command description - since - debut version - group - command group - complexity - complexity string - doc-flags - flags used for documentation (e.g. "deprecated") - deprecated-since - if deprecated, from which version? - replaced-by - if deprecated, which command replaced it? - history - a list of (version, what-changed) tuples - hints - a list of strings, meant to provide hints for clients/proxies. see #9876 - arguments - an array of arguments. each element is a map, with the possibility of nesting (sub-arguments) - key-specs - an array of keys specs (already in unstable, just changed location) - subcommands - a list of sub-commands (already in unstable, just changed location) - reply-schema - will be added in the future (see #9845) more details on these can be found in redis/redis-doc#1697 only the first three fields are mandatory #### API changes (unreleased API obviously) now they take RedisModuleCommand opaque pointer instead of looking up the command by name - RM_CreateSubcommand - RM_AddCommandKeySpec - RM_SetCommandKeySpecBeginSearchIndex - RM_SetCommandKeySpecBeginSearchKeyword - RM_SetCommandKeySpecFindKeysRange - RM_SetCommandKeySpecFindKeysKeynum Currently, we did not add module API to provide additional information about their commands because we couldn't agree on how the API should look like, see #9944. ### Somehow related changes 1. Literals should be in uppercase while placeholder in lowercase. Now all the GEO* command will be documented with M|KM|FT|MI and can take both lowercase and uppercase ### Unrelated changes 1. Bugfix: no_madaory_keys was absent in COMMAND's reply 2. expose CMD_MODULE as "module" via COMMAND 3. have a dedicated uint64 for ACL categories (instead of having them in the same uint64 as command flags) Co-authored-by: Itamar Haber <itamar@garantiadata.com>

Delete the hardcoded command table and replace it with an auto-generated table, based on a JSON file that describes the commands (each command must have a JSON file). These JSON files are the SSOT of everything there is to know about Redis commands, and it is reflected fully in COMMAND INFO. These JSON files are used to generate commands.c (using a python script), which is then committed to the repo and compiled. The purpose is: * Clients and proxies will be able to get much more info from redis, instead of relying on hard coded logic. * drop the dependency between Redis-user and the commands.json in redis-doc. * delete help.h and have redis-cli learn everything it needs to know just by issuing COMMAND (will be done in a separate PR) * redis.io should stop using commands.json and learn everything from Redis (ultimately one of the release artifacts should be a large JSON, containing all the information about all of the commands, which will be generated from COMMAND's reply) * the byproduct of this is: * module commands will be able to provide that info and possibly be more of a first-class citizens * in theory, one may be able to generate a redis client library for a strictly typed language, by using this info. ### Interface changes #### COMMAND INFO's reply change (and arg-less COMMAND) Before this commit the reply at index 7 contained the key-specs list and reply at index 8 contained the sub-commands list (Both unreleased). Now, reply at index 7 is a map of: - summary - short command description - since - debut version - group - command group - complexity - complexity string - doc-flags - flags used for documentation (e.g. "deprecated") - deprecated-since - if deprecated, from which version? - replaced-by - if deprecated, which command replaced it? - history - a list of (version, what-changed) tuples - hints - a list of strings, meant to provide hints for clients/proxies. see redis#9876 - arguments - an array of arguments. each element is a map, with the possibility of nesting (sub-arguments) - key-specs - an array of keys specs (already in unstable, just changed location) - subcommands - a list of sub-commands (already in unstable, just changed location) - reply-schema - will be added in the future (see redis#9845) more details on these can be found in redis/redis-doc#1697 only the first three fields are mandatory #### API changes (unreleased API obviously) now they take RedisModuleCommand opaque pointer instead of looking up the command by name - RM_CreateSubcommand - RM_AddCommandKeySpec - RM_SetCommandKeySpecBeginSearchIndex - RM_SetCommandKeySpecBeginSearchKeyword - RM_SetCommandKeySpecFindKeysRange - RM_SetCommandKeySpecFindKeysKeynum Currently, we did not add module API to provide additional information about their commands because we couldn't agree on how the API should look like, see redis#9944. ### Somehow related changes 1. Literals should be in uppercase while placeholder in lowercase. Now all the GEO* command will be documented with M|KM|FT|MI and can take both lowercase and uppercase ### Unrelated changes 1. Bugfix: no_madaory_keys was absent in COMMAND's reply 2. expose CMD_MODULE as "module" via COMMAND 3. have a dedicated uint64 for ACL categories (instead of having them in the same uint64 as command flags) Co-authored-by: Itamar Haber <itamar@garantiadata.com>

Co-authored-by: Oran Agra <oran@redislabs.com>

oranagra

only reviewed the tips.
slightly changed the meaning of all_succeeded and one_succeeded.
taking the liberty to apply my changes to make it easier for future reviewers.

topics/command-tips.md

oranagra

I haven't had time to review comand-arguments.md and key-specs.md.
i trust that the information there is ok as a first version, so if someone else reviewed it, we can merge this PR.
and even if no one else reviewed it, i rather merge it as is than keep it waiting (we can always incrementally improve it in future PRs)
@itamarhaber ?

commands/command.md

oranagra · 2022-01-17T12:23:59Z

commands/command.md

+ - nested @array-reply of [ACL categories][ta]
+ - nested @array-reply of [command tips][tb]
+ - nested @array-reply of [key-specs][td]
+ - nested @array-reply of subcommands
+[ta]: /topics/acl
+[tb]: /topics/command-tips
+[td]: /topics/key-specs


no sure about this markdown, is it sane?

@itamarhaber ?

The syntax is ok, but there should be a \n before these just for safety (i.e. regular md is ok with that, but the current site's implementation is more sensitive).

@itamarhaber what do you mean? an empty line between the - nested list and the [ta] thing?
i must admit i don't understand any of this.. what does it do?

anyway, and you add a suggestion to fix it, or even just push a fix?

Yes, will push the fixes.

commands/command.md

topics/command-arguments.md

topics/command-tips.md

oranagra

ok with me..
i think the last remaining real issue is to decide if we really want to remove block_keyword

topics/command-tips.md

oranagra · 2022-01-20T09:33:15Z

@itamarhaber the redis PR was merged.
please merge when ready (and then do some sanity check on the live version)

oranagra · 2022-01-21T11:01:31Z

@itamarhaber it is impossible to review your last change, can you please sum up what you changed?
is it mainly about wording? also added / removed some information? moved things around?

itamarhaber · 2022-01-23T15:13:46Z

@oranagra edits are grammatic, syntactic, and formatting mostly.

No content has been intentionally added/removed - I think it can be merged.

oranagra · 2022-01-23T15:28:08Z

ok. go merge.
just please confirm that it has all the recent changes (if applicable):
redis/redis#10147
redis/redis#10104
redis/redis#10127
redis/redis#10122
redis/redis#10056

itamarhaber · 2022-01-23T17:19:14Z

@oranagra thanks for the list - reviewed.

Improved handling of subcommands (don't allow ACL on first-arg of a sub-command) redis#10147 - not applicable imo
Add command tips to COMMAND DOCS redis#10104 - applicable and included (COMMAND and new topic)
sub-command support for ACL CAT and COMMAND LIST. redisCommand always stores fullname redis#10127 - applicable and included (COMMAND)
New detailed key-spec flags (RO, RW, OW, RM, ACCESS, UPDATE, INSERT, DELETE) redis#10122 - applicable and included (COMMAD and two new topics)
Move doc metadata from COMMAND to COMMAND DOCS redis#10056 - applicable and included (COMMAD)

The only thing that bothers me a little is that we have new dangling topics (topics/command-tips, topics/key-specs, topics/command-arguments) without a reference from the documentation view of the website (separate PR needed). I'm not sure it is actually needed, but if so I was thinking about adding Command runtime introspection section after the cluster's.

oranagra · 2022-01-23T17:35:12Z

@itamarhaber i do agree we need to bind these topics somehow in the general docs section, same as we should for "programability".
feel free to merge this one and do that in another PR.

Refresh COMMAND, add topics: command-args, key-specs

62e48ff

guybe7 force-pushed the command_doc branch from f129d52 to 62e48ff Compare December 1, 2021 15:12

guybe7 mentioned this pull request Dec 1, 2021

Auto-generate the command table from JSON files redis/redis#9656

Merged

11 tasks

oranagra reviewed Dec 1, 2021

View reviewed changes

guybe7 added 5 commits December 3, 2021 16:27

CR fixes

3841e8e

document key-spec-index

c2339a2

spelling

52f80d6

spelling 2

7b5908f

add COMMAND LIST to commands.json

a9cf8d5

itamarhaber reviewed Dec 4, 2021

View reviewed changes

oranagra reviewed Dec 5, 2021

View reviewed changes

topics/command-info.md Outdated Show resolved Hide resolved

guybe7 and others added 2 commits December 5, 2021 10:17

Apply suggestions from code review

bcee2b6

Co-authored-by: Itamar Haber <itamar@redis.com>

Update commands/command.md

92e168e

Co-authored-by: Itamar Haber <itamar@redis.com>

guybe7 added to-be-merged should probably be merged soon waiting-for-upstream waiting for a redis PR to be merged labels Dec 15, 2021

key specs

6b18e58

oranagra reviewed Dec 15, 2021

View reviewed changes

topics/key-specs.md Outdated Show resolved Hide resolved

topics/key-specs.md Outdated Show resolved Hide resolved

topics/key-specs.md Outdated Show resolved Hide resolved

itamarhaber mentioned this pull request Dec 16, 2021

Replaces commands.json with new syntax #1714

Merged

2 tasks

guybe7 and others added 7 commits January 7, 2022 10:26

Apply suggestions from code review

6393988

Co-authored-by: Oran Agra <oran@redislabs.com>

Merge remote-tracking branch 'origin/master' into command_doc

e469d6c

adjust to COMMAND DETAILS

7179530

refresh cmd flags + spelling

a8b9012

update commands.json + rename DETAILS to DOCS

f995dac

command tips

a848d83

add exmaple

9f77ec4

guybe7 mentioned this pull request Jan 12, 2022

Update COMMAND and ACL docs #1669

Closed

oranagra reviewed Jan 12, 2022

View reviewed changes

edit tips.md

f7e5f40

guybe7 added 2 commits January 13, 2022 12:04

fix

7a3a87b

fix 2

734abaf

oranagra reviewed Jan 17, 2022

View reviewed changes

yossigo reviewed Jan 17, 2022

View reviewed changes

yossigo mentioned this pull request Jan 17, 2022

Add command tips to COMMAND DOCS redis/redis#10104

Merged

guybe7 and others added 3 commits January 17, 2022 16:23

fixes 3

26cd600

fix 4

70ddd01

remove the term proxy

5bbc7b3

oranagra approved these changes Jan 17, 2022

View reviewed changes

madolson reviewed Jan 18, 2022

View reviewed changes

updating the new keys-spec flags

47423a5

guybe7 mentioned this pull request Jan 18, 2022

Define and document command tips redis/redis#9876

Closed

apply latest decisions

8392095

oranagra approved these changes Jan 19, 2022

View reviewed changes

align with the changes in the redis pr

83d4a45

oranagra removed the waiting-for-upstream waiting for a redis PR to be merged label Jan 20, 2022

Applies some edits.

0f708b6

More edits.

a595bc6

Fixes typo.

a1e8599

oranagra merged commit 06f3598 into redis:master Jan 24, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refresh COMMAND, add topics: command-args, key-specs #1697

Refresh COMMAND, add topics: command-args, key-specs #1697

guybe7 commented Dec 1, 2021 •

edited

Loading

oranagra left a comment

oranagra left a comment

oranagra left a comment

oranagra Jan 17, 2022

guybe7 Jan 17, 2022

itamarhaber Jan 19, 2022

oranagra Jan 19, 2022

itamarhaber Jan 19, 2022

oranagra left a comment

oranagra commented Jan 20, 2022

oranagra commented Jan 21, 2022

itamarhaber commented Jan 23, 2022 •

edited

Loading

oranagra commented Jan 23, 2022

itamarhaber commented Jan 23, 2022

oranagra commented Jan 23, 2022

Refresh COMMAND, add topics: command-args, key-specs #1697

Refresh COMMAND, add topics: command-args, key-specs #1697

Conversation

guybe7 commented Dec 1, 2021 • edited Loading

oranagra left a comment

Choose a reason for hiding this comment

oranagra left a comment

Choose a reason for hiding this comment

oranagra left a comment

Choose a reason for hiding this comment

oranagra Jan 17, 2022

Choose a reason for hiding this comment

guybe7 Jan 17, 2022

Choose a reason for hiding this comment

itamarhaber Jan 19, 2022

Choose a reason for hiding this comment

oranagra Jan 19, 2022

Choose a reason for hiding this comment

itamarhaber Jan 19, 2022

Choose a reason for hiding this comment

oranagra left a comment

Choose a reason for hiding this comment

oranagra commented Jan 20, 2022

oranagra commented Jan 21, 2022

itamarhaber commented Jan 23, 2022 • edited Loading

oranagra commented Jan 23, 2022

itamarhaber commented Jan 23, 2022

oranagra commented Jan 23, 2022

guybe7 commented Dec 1, 2021 •

edited

Loading

itamarhaber commented Jan 23, 2022 •

edited

Loading