Use RM_BlockedClientMeasureTimeStart and RM_BlockedClientMeasureTimeEnd on async executions #586
Comments
MeirShpilraien
added a commit
that referenced
this issue
Aug 22, 2021
…asureTimeEnd on async execution to get slowlog report
MeirShpilraien
added a commit
that referenced
this issue
Aug 24, 2021
…asureTimeEnd on async execution to get slowlog report (#592) * see #586, use RM_BlockedClientMeasureTimeStart and RM_BlockedClientMeasureTimeEnd on async execution to get slowlog report * return accidently deleted GetNullRecord * fix failing tests * unified callback of running and holding with execution done * added tests to verify that async executions are getting into the slowlog
MeirShpilraien
added a commit
that referenced
this issue
Aug 24, 2021
…lientMeasureTimeEnd on async execution to get slowlog report
Merged
Merged
gkorland
added a commit
that referenced
this issue
Jan 31, 2022
* Fix crash on cluster set from shard (#506) * Fix bad test (#510) * use mkdocs-modules-template (#516) * upgrade mkdocs & mkdocs-material (#517) * adding gmake4 to centos7 for RedisGears This is not yet wired up to CI * Decrease ref count of value after appending it to list * Moved to Redis 6.2 (#519) * Fix valgrind failure on Redis 6.2 (#526) * Fix valgrind failure on Redis 6.2 * More fixes * Build updates (#530) * Add json example (#540) * added json example to docs * added json-avg.py * Update examples.md Co-authored-by: Guy Korland <gkorland@gmail.com> * Added redgrease as client library to the docs (#543) * fix unstable test (#544) * fix unstable test * initialize runId to NULL * increase max idle time to avoid failures on mac * Fix tests * Update config.yml (#547) * make platform + build/docker REDIS_VER (#548) * fix #554, fix config get to return user defined config (#557) * Avoid warnings from redisgears.h (#555) * fix unstable valgrind test (#556) * fix unstable valgrind test * review fixes * early returns (#559) * updating to redis 6.2.4 (#567) * Added QA automation tests (#531) * Build system changes (#571) * Lock Redis GIL when creating RedisAI DAG from string (#572) * build/docker repo name fix (#573) * removing docker curl and executing, relying on the contents of github… (#575) * Add null record type (#578) * Add null record type * make NullRecord singelton * remove special record * init NullRecord * update apis * make NullRecord a special record * rename IS_SPECIAL_RECORD to IS_SINGLETON * Change package name scheme (#579) * release drafter and release actions (#583) * Update Redis Module API; support for GCC 10 and Ubuntu/Hirsute (2) (#582) * added inorder option to command reader (#587) * added inorder option to command reader * free wd when dropping the registration * added check for rdbsave with inorder * review fixes * fix test * Added python-dev to system-setup (#589) * fix fluky tests (#593) * see #586, use RM_BlockedClientMeasureTimeStart and RM_BlockedClientMeasureTimeEnd on async execution to get slowlog report (#592) * see #586, use RM_BlockedClientMeasureTimeStart and RM_BlockedClientMeasureTimeEnd on async execution to get slowlog report * return accidently deleted GetNullRecord * fix failing tests * unified callback of running and holding with execution done * added tests to verify that async executions are getting into the slowlog * fix test failure on redis version older than 6.2 (#596) * added RedisGears info section to redis info command (#594) * added RedisGears info section to redis info command * added plugin stats * added test to info command * fix valgrind failure (#597) * fix mac failure (#601) * fix MOD-1676, check reply type instead of assert it (#609) * Docker/debian: moved from Buster to Bullseye (#608) * fix issue where stream reader might trigger executions on replica (#613) * fix issue where stream reader might trigger executions on replica * Fix memory leak * Fixed typo in comment. (#614) Co-authored-by: Guy Korland <gkorland@gmail.com> * Grammar fix, Labs removed. (#607) Co-authored-by: Guy Korland <gkorland@gmail.com> * Adding proper path to python extras (#605) * update python interpreter version to 3.7.12 (#624) * update python interpreter version to 3.7.12 * python allocator will return 16-byte aligned addresses * added MOD-1960 test to master branch (#630) * Bump mkdocs from 1.1.2 to 1.2.3 in /docs (#625) Bumps [mkdocs](https://github.com/mkdocs/mkdocs) from 1.1.2 to 1.2.3. - [Release notes](https://github.com/mkdocs/mkdocs/releases) - [Commits](mkdocs/mkdocs@1.1.2...1.2.3) --- updated-dependencies: - dependency-name: mkdocs dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Guy Korland <gkorland@gmail.com> * Update README.md (#628) Co-authored-by: Chayim <chayim@users.noreply.github.com> * tests fixes after redispy upgrade (#635) * added commands.json (#634) * Redis AI documentation (#622) * Address Chayim comments for proper english * Fix valgrind failures (#638) * Upgrade pytest version to 6.2.5 (#639) * Add redisai docs to nav bar and home page (#641) * Cosmetic fixes for AI docs (#642) * added python profiler when ProfileExecutions is enabled (#603) * added python profiler when ProfileExecutions is enabled * release session iterator * added profiler tests * tests * update docs * fix tests * review fixes * changing O2 to O3 in makefile * Fix #558, wait for cluster to be initialized when reading stream data (#656) On oss cluster, if cluster is not initialized, read commands will failed and return an error. On stream reader, we assert that we read the stream successfully, such assertion will failed on uninitialized cluster and will cause a crash. Solution: 1. Before start scaning for streams, wait for cluster to be initialized. 2. '1' is not enough, by the time we find a stream to read the cluster might get uninitialized again. To solve this, if we failed to read the stream we retry after 5 seconds. We retry up to 10 times and than abort. * Random port on test nextwork (#660) * Random port on test nextwork After merging this RedisLabsModules/RLTest#147 it will be possible to run the tests in parallel but for this to work we need a random port to avoid collisions. * temporarly use parallel test branch * remove old rltest * remove old rltest * remove old rltest * reduce parallelism * remove reinstallation of rltest * Adds 'Plugin' module configuration parameter (#560) Co-authored-by: Meir Shpilraien (Spielrein) <meir@redislabs.com> Co-authored-by: Meir Shpilraien (Spielrein) <meir@redis.com> * Add AdGears to the list of demos (#564) AdGears is another demo around RedisGears Co-authored-by: Meir Shpilraien (Spielrein) <meir@redislabs.com> Co-authored-by: Meir Shpilraien (Spielrein) <meir@redis.com> * Valgrind fixes (#662) * Valgrind fixes: 1. Fix wrong valgrind error when writing pointer address to pipe. Valgrind might think we lost the pointer if the processes exit before reading it from the pipe. Added this case to suppression file. 2. Use shutdown event to do cleanup instead of __attribute__((destructor)) This way we know that we run when the GIL help and there will be no race conditions on cleanups. * remove unneeded lockhandler release * Use safe lock in RedisAI initialization (#667) * Used python decorator to perform cleanups after tests (#637) * WIP, keep env between tests * small copy past fix * fixes to run with reuse env * additional small fixes * avoid cleanup if reuse-env is not used * perform cleanup anyway to avoid valgrind errors Co-authored-by: Chayim <chayim@users.noreply.github.com> * fix mac and valgrind failures by increase cluster-node-timeout to 60 seconds. (#670) * fix fluky test, stream reader creates more than one execution on a stream. (#671) * Fix #650, do not propagate multi exec on Redis 7. (#664) This PR is for future compatability to Redis 7. Till Redis 7, if commands would have executed from background thread, then Redis would not have wrap them with multi exec and the commands would not have executed atomically on replica. Gears solved it by propagating multi exec by itself. Thanks to this PR: redis/redis#9890 It is no longer needed to replicate multi exec, Redis takes care of this for us. * Fix #472, when loading RDB generated by version 1.0, check that python plugin is loaded. (#663) RDB generated by version 1.0 needs the python plugin in order to be loaded successfully. Failed the loading gracefully if the python plugin is missing. * Fix #536, added RG.TRIGGERONKEY (#661) * Fix #536, added RG.TRIGGERONKEY The **RG.TRIGGERONKEY** command is the same as **RG.TRIGGER**, the difference is that the third argument is considered a key so client will know how to direct the command to the correct shard. **Redis API** ``` RG.TRIGGERONKEY <trigger> key [arg ...] ``` _Arguments_ * **trigger**: the trigger's name * **key**: a key on which the trigger is executed on * **arg**: any additional arguments _Return_ An array containing the function's output records. **Examples** ``` redis> RG.PYEXECUTE "GB('CommandReader').map(lambda x: execute('set', x[1], x[2])).register(trigger='my_set')" OK redis> RG.TRIGGERONKEY my_set foo bar 1) "OK" redis> get foo bar ``` * fix test * fix fluky tests * review fixes * Fix #619, use `getkeys-api` flag to allow Redis to catch MOVED error. (#673) * Fix #619, use `getkeys-api` flag to allow Redis to catch MOVED error. Hooking a command works using a command filter that direct a command to another internal command that runs the hook code. So for example, if the use sends `get x`, Redis will actually see `rg.trigger get x`. Redis do not know that on `rg.trigger get x`, `x` is actually a key and it will not send MOVED error on cluster. The fix, use `getkeys-api` to allow to tell Redis dynamically where keys are. When `rg.trigger get x` is invoke, Redis will ask RedisGears to tell it where are the keys in the command and gears will dynamically calculate the keys base on the original command that was executed. Tests was added to verify the fix. * Fix tests * Fix leak on CommandHook_DeclareKeys (#677) The keys value returned from RedisModule_GetCommandKeys was not freed * Fix #665, allow case insensetive event type on command reader. (#679) * tying docs builds to 3.9 until mkdocs is upgraded (#621) Co-authored-by: Guy Korland <gkorland@gmail.com> * Fix globals dictionary not set correctly after deserialization. (#676) The problem --- After deserialization, the deserialized function are created with its own globals dictionary. We want all the functions on the same execution (and even different execution that was generated by the same `RG.PYEXECUTE` command) to share the same globals dictionary. The current solution is to change the function global dictionary to be the global dictionary of the `PY.EXECUTE` session that created the execution. This solution only changes the top most function, but it does not change function that is used by it, so the following example will break: ``` g = 1 def f1(): global g g = g + 1 print(id(globals())) return g def f(x): global g g = g + 1 print(id(globals())) return f1() GB('CommandReader').map(f).register(trigger='test', convertToStr=False) GB('CommandReader').map(f).register(trigger='test1', convertToStr=False) ``` Running `RG.TRIGGER test` and `RG.TRIGGER test1` will give the following: ``` > RG.TRIGGER test 3 > RG.TRIGGER test1 5 ``` But after RDB load we will get the following result (that proves that `f1` was created with a different global dictionary): ``` > RG.TRIGGER test 3 > RG.TRIGGER test1 4 ``` The solution --- Modify `cloudpickle` to look for a global variable, `requested_base_globals`. If set, use this dictionary as the global dictionary of all the created functions. RedisGears will make sure to set `requested_base_globals` to the correct value before deserializing a function. In addition, changed `cloudpickle` to update the global dictionary only with the missing fields and not override existing fields. After this change, the above example works as expected. Test was added to verify the fix. * Documentation (#518) * document async await support * fix tests * allow await inside sync execution * added Async Await Advance Topics to docs * added async_await-004.py * added async_await_advance_topics.md * document command hook * fix docs tests * spellcheck fixes * added all the examples to docs tests * added key miss tutorial and document new arguments to command reader * added small details to override_reply docs * pr comments Co-authored-by: Chayim I. Kirshen <c@kirshen.com> * small doc fixes (#680) * Fix typo (#681) * Extract more information on test failure (#682) * Extract more information on test failure Extract the following information of test failure: 1. RG.DUMPREGISTRATIONS 2. RG.DUMPEXECUTIONS 3. info everything The information will be written to a file on the log directory. File name will be the same as the Redis log file name with a suffix contains the test name. This will promise that the collected informatio will be uploaded to the CI and we will be able to analize it. * generate the information on any test failure * first pass to build jvm (#675) Co-authored-by: Meir Shpilraien (Spielrein) <meir@redis.com> * Adding JVMPlugin Directly (#683) * snapshots do not depend on osx now (#685) * Added information to RG.DUMPREGISTRATIONS (#684) * Added information to RG.DUMPREGISTRATIONS The new information is: 1. lastRunDurationMS - time (in MS) of the last execution 2. totalRunDurationMS - total runtime in MS 3. avgRunDurationMS - average execution time in MS Also, for stream reader, added the following information: 1. lastEstimatedLagMS - last batch lag (diff from the momenent the first batch entry got to the stream to the moment the batch processing was finished). 2. avgEstimatedLagMS - average lag of all batches. * added rg.clearregistrationsstats * fix jvm tests * review fixes * rename redislabs redis (#687) * Update examples.md * Update examples.md * Fixing artifact s3 push (#689) * Fix #545, check REDISMODULE_CTX_FLAGS_DENY_BLOCKING flag before blocking the client. (#688) * Fix #545, check REDISMODULE_CTX_FLAGS_DENY_BLOCKING flag before blocking the client. We check it on 3 places and act as if we run in MULTI/EXEC or Lua script: 1. RG.PYEXECUTE 2. Command override 3. Keys Reader commands option * skip tests on redis 6.0 * review fixes * fix tests * jvm snapshots not snapshot (#695) * Support for forked pull requests (#696) * Fix hashtag() function on enterprise. (#697) * Update min_redis_pack_version to 6.0.12 (#698) * Add java client to docs (#692) Co-authored-by: Meir Shpilraien (Spielrein) <meir@redis.com> Co-authored-by: Guy Korland <gkorland@gmail.com> Co-authored-by: Chayim I. Kirshen <c@kirshen.com> Co-authored-by: Chayim <chayim@users.noreply.github.com> * publishing the jvmplugin to maven (#700) * See #693, support for session names (session id) and upgrades. (#699) See #693, support for session names (session id) and upgrades. Following greate feedback from a lot of uses, the following PR make it easy and safer to update registrations code. Current upgrade process: * Unregister all relevant registrations * Send the new code Problems with the current process: * Registrations have none informative ID, it hard to find the registrations that need to be removed. Most of the users put a common description to find relevant registrations but its still require a hard and frustrated manual process. * The process is not atomic, which means that you must stop your traffic during this process otherwise you might get data while there is no registrations. Some recipe attempt to make it atomic and have the new code unregistered the old code (look at [rgsync](https://github.com/RedisGears/rgsync) and [rghibernate](https://github.com/RedisGears/rghibernate) for a reference). Thought this approach makes the process simpler, it is still unsafe on cluster because on the other nodes the unregistered/register will not happened atomically. * There is no way to pass data between the old version and the new version of the code, if the code created new registrations base on user input there is not way for the new code to know that. The following PR attempt to solve the above problems. The PR define a new entity called session, a new session is created whenever a user performs RG.PYEXECUTE (or RG.JEXECUTE for java), all the registrations that created (and will be created in the future) by this RG.PYEXECUTE will belong to the created session. Sessions can be named with a unique ID, so it will be easy to find your code and all the registrations it created using either, `RG.PYDUMPSESSION` or `RG.JDUMPSESSIONS` for python and java respectively. Moreover it is possible to easily update your code, RedisGears will track all the registration created by your code and upon upgrade, will first unregistered all the old registrations and only then run your new code and allow it to create new registrations. RedisGears will make this entire process atomically on all the shards and will make sure to revert the entire process on failure (so if the new code fails for some reason your old registrations stay untouched). We will describe the API to achieve the above for both Python and JVM plugin. # Python API Test was added under test_session.py, to test the new API. ## `RG.PYEXECUTE` Changes `RG.PYEXECUTE` command was changed as following: ``` RG.PYEXECUTE <code> [UNBLOCKING] [ID <id>] [DESCRIPTION <description>] [UPGRADE] [REPLACE_WITH id] [REQUIREMENTS "<dep> ..."] ``` REQUIREMENTS is the only argument that must come last. The other arguments are optional. New arguments explanation: * ID - A unique id of the session. If not given, RedisGears will generate a random id. * DESCRIPTION - Text description of what the code does, will be printed on `RG.PYDUMPSESSIONS`. * UPGRADE - If the given id already exists and error will be raise unless the `UPGRADE` argument is given and direct RedisGears to upgrade the existing code. * REPLACE_WITH - In case you want to give a new ID and replace it with some other session, this argument is so it will be possible to upgrade v1 registration on which the ID option was not exists. Example: ``` 127.0.0.1:6379> RG.PYEXECUTE "GB('CommandReader').map(lambda x: 'OK').register(trigger='test')" ID test OK 127.0.0.1:6379> RG.TRIGGER test 1) "OK" 127.0.0.1:6379> RG.PYEXECUTE "GB('CommandReader').map(lambda x: 'new value').register(trigger='test')" ID test (error) Session test already exists 127.0.0.1:6379> RG.PYEXECUTE "GB('CommandReader').map(lambda x: 'new value').register(trigger='test')" ID test UPGRADE OK 127.0.0.1:6379> RG.TRIGGER test 1) "new value" ``` ## `RG.PYDUMPSESSIONS` Changes `RG.PYDUMPSESSIONS` command was changed as following: ``` RG.PYDUMPSESSIONS [DEAD] [VERBOSE] [SESSIONS s1 s2 ...] ``` `RG.PYDUMPSESSIONS` command will now provide more details about the session, arguments explanation: * VERBOSE - see a full information about requirements and registrations. * DEAD - see session which was deleted but not yet freed (because there is still executions which created by the session and was not yet finished). * SESSIONS - must be given last. If given, return only sessions that appears in the given list. Example: ``` 127.0.0.1:6379> RG.PYDUMPSESSIONS 1) 1) "ID" 1) "test" 2) "sessionDescription" 3) (nil) 4) "refCount" 5) (integer) 1 6) "Linked" 7) "true" 8) "dead" 1) "false" 2) "requirementInstallationNeeded" 3) (integer) 0 4) "requirements" 5) (empty array) 6) "registrations" 7) 1) "0000000000000000000000000000000000000000-3" 127.0.0.1:6379> RG.PYDUMPSESSIONS VERBOSE SESSIONS test 1) 1) "ID" 2) "test" 3) "sessionDescription" 4) (nil) 5) "refCount" 6) (integer) 1 7) "Linked" 8) "true" 9) "dead" 10) "false" 11) "requirementInstallationNeeded" 12) (integer) 0 13) "requirements" 14) (empty array) 15) "registrations" 16) 1) 1) "id" 2) "0000000000000000000000000000000000000000-3" 3) "reader" 4) "CommandReader" 5) "desc" 6) (nil) 7) "RegistrationData" 8) 1) "mode" 2) "async" 3) "numTriggered" 4) (integer) 1 5) "numSuccess" 6) (integer) 1 7) "numFailures" 8) (integer) 0 9) "numAborted" 10) (integer) 0 11) "lastRunDurationMS" 12) (integer) 0 13) "totalRunDurationMS" 14) (integer) 0 15) "avgRunDurationMS" 16) "0" 17) "lastError" 18) (nil) 19) "args" 20) 1) "trigger" 2) "test" 3) "inorder" 4) (integer) 0 9) "ExecutionThreadPool" 10) "DefaultPool" ``` The above information was also added to redis info command under `rg` section, example: ``` 127.0.0.1:6379> info rg # rg rg_nexecutions:0 rg_nregistrations:1 # rg_regisrations rg_0000000000000000000000000000000000000000-3:desc=None,reader=CommandReader,mode=async,numTriggered=1,numSuccess=1,numFailures=0,numAborted=0,lastRunDurationMS=0,totalRunDurationMS=0,avgRunDurationMS=0,lastError=None,trigger=test,inorder=0,PD={'sessionName':'test'| 'sessionDescription':'null'| 'refCount': 1| 'linked': true| 'dead': false| 'isInstallationNeeded':0| 'registrationsList':['0000000000000000000000000000000000000000-3']| 'depsList':[]} # rg_plugins rg_GearsPythonPlugin:version=10000 # rg_python_stats rg_TotalAllocated:23742392 rg_PeakAllocated:8001216 rg_CurrAllocated:7945584 # rg_python_requirements # rg_python_sessions rg_test:{'sessionName':'test', 'sessionDescription':'null', 'refCount': 1, 'linked': true, 'dead': false, 'isInstallationNeeded':0, 'registrationsList':['0000000000000000000000000000000000000000-3'], 'depsList':[]} ``` ## Upgrade Limitation Upgrading your python code will **not** upgrade your requirement, the python interpreter already loaded the requirements code into the memory and changing them on the file system will not help. Currently upgrade requirement requires restart. For more information about this topic please refer to [Isolation Technics](https://oss.redis.com/redisgears/1.0/isolation_technics.html) page. # JVM API ## `RG.JEXECUTE` Changes `RG.JEXECUTE` command was changed as following: ``` RG.JEXECUTE <path.to.main.class> [UPGRADE] [SKIP_VERSION_CHECK] <jar payload> ``` On JVM plugin, the session id is the <path.to.main.class> also the `UPGRADE` argument can be given to indicate that we want to update the existing code. example: Given the following code: ```java package gears_experiments; import gears.GearsBuilder; import gears.readers.CommandReader; public class test { public static void main(String[] args) { GearsBuilder.CreateGearsBuilder(new CommandReader().setTrigger("test")).map(r->"OK").register(); } } ``` Register the code to RedisGears with the JVM plugin: ``` > redis-cli -x RG.JEXECUTE gears_experiments.test < ./test.jar "OK" > redis-cli RG.TRIGGER test 1) "OK" ``` Trying to register it again will raise an error indicating that the session already exists: ``` > redis-cli -x RG.JEXECUTE gears_experiments.test < ./test.jar (error) Session gears_experiments.test already exists ``` Register with the `UPGRADE` argument will successed: ``` > redis-cli -x RG.JEXECUTE gears_experiments.test UPGRADE < ./test.jar OK ``` ### JVM Description and Version The JVM Plugin allows to create a static `String` variable on the main class called `DESCRIPTION`. The value of this field will be considered as the session description. In addition, JVM Plugin allows to create a static `int` variable on the main class called `VERSION`, the value of this field will be considered as the session version. To prevent mistakes on upgrades, RedisGears will enforce versioning and will only allow upgrade to a newer version (unless `SKIP_VERSION_CHECK` argument is give to `RG.JEXECUTE` command). If the `VERSION` field is not given or set to `-1`, always allow upgrade. Example: The folloing code will have description and version: ```java package gears_experiments; import gears.GearsBuilder; import gears.readers.CommandReader; public class test { public static String DESCRIPTION = "foo"; public static int VERSION = 1; public static void main(String[] args) { GearsBuilder.CreateGearsBuilder(new CommandReader().setTrigger("test")).map(r->"OK").register(); } } ``` Register the code: ``` > redis-cli -x RG.JEXECUTE gears_experiments.test < ./test.jar OK > redis-cli RG.JDUMPSESSIONS 1) 1) "mainClass" 2) "gears_experiments.test" 3) "version" 4) (integer) 1 5) "description" 6) "foo" 7) "upgradeData" 8) (nil) 9) "jar" 10) "/home/meir/work/RedisGears/plugins/jvmplugin/-jars/6876b8b78ccfc2ad764edc7ede590f573bd7260b.jar" 11) "refCount" 12) (integer) 2 13) "linked" 14) "true" 15) "dead" 16) "false" 17) "registrations" 18) 1) "0000000000000000000000000000000000000000-1" ``` We can see (from `RG.JDUMPSESSIONS` output) that the version is 1 and the description is `foo`. Trying to upgrade the code now will raise an error: ``` > redis-cli -x RG.JEXECUTE gears_experiments.test UPGRADE < ./test.jar (error) Session with higher (or equal) version already exists, current version is 1 and new version is 1 ``` ## `RG.JDUMPSESSIONS` Changes Changed to `RG.JDUMPSESSIONS` are similar to `RG.PYDUMPSESSIONS`. Output example: ``` > redis-cli RG.JDUMPSESSIONS 1) 1) "mainClass" 2) "gears_experiments.test" 3) "version" 4) (integer) 1 5) "description" 6) "foo" 7) "upgradeData" 8) (nil) 9) "jar" 10) "/home/meir/work/RedisGears/plugins/jvmplugin/-jars/6876b8b78ccfc2ad764edc7ede590f573bd7260b.jar" 11) "refCount" 12) (integer) 2 13) "linked" 14) "true" 15) "dead" 16) "false" 17) "registrations" 18) 1) "0000000000000000000000000000000000000000-1" > redis-cli RG.JDUMPSESSIONS VERBOSE SESSIONS gears_experiments.test 1) 1) "mainClass" 2) "gears_experiments.test" 3) "version" 4) (integer) 1 5) "description" 6) "foo" 7) "upgradeData" 8) (nil) 9) "jar" 10) "/home/meir/work/RedisGears/plugins/jvmplugin/-jars/6876b8b78ccfc2ad764edc7ede590f573bd7260b.jar" 11) "refCount" 12) (integer) 2 13) "linked" 14) "true" 15) "dead" 16) "false" 17) "registrations" 18) 1) 1) "id" 2) "0000000000000000000000000000000000000000-1" 3) "reader" 4) "CommandReader" 5) "desc" 6) (nil) 7) "RegistrationData" 8) 1) "mode" 2) "async" 3) "numTriggered" 4) (integer) 0 5) "numSuccess" 6) (integer) 0 7) "numFailures" 8) (integer) 0 9) "numAborted" 10) (integer) 0 11) "lastRunDurationMS" 12) (integer) 0 13) "totalRunDurationMS" 14) (integer) 0 15) "avgRunDurationMS" 16) "-nan" 17) "lastError" 18) (nil) 19) "args" 20) 1) "trigger" 2) "test" 3) "inorder" 4) (integer) 0 9) "ExecutionThreadPool" 10) "JVMPool" ``` ## `info rg` Changes JVM session information was added to `info rg` along side the JVM memory stats. Output example: ``` > redis-cli info rg # rg rg_nexecutions:0 rg_nregistrations:1 # rg_regisrations rg_0000000000000000000000000000000000000000-1:desc=None,reader=CommandReader,mode=async,numTriggered=0,numSuccess=0,numFailures=0,numAborted=0,lastRunDurationMS=0,totalRunDurationMS=0,avgRunDurationMS=-nan,lastError=None,trigger=test,inorder=0,PD={'MainClassName': 'gears_experiments.test'| 'Version': 1| 'Description': 'foo'| 'UpgradeData': 'Null'| 'Linked': 'true'| 'dead': 'false'| 'JarFilePath': '/home/meir/work/RedisGears/plugins/jvmplugin/-jars/6876b8b78ccfc2ad764edc7ede590f573bd7260b.jar'| 'Registrations': '['0000000000000000000000000000000000000000-1']'} # rg_plugins rg_GearsJVMPlugin:version=999999 # rg_jvm_sessions rg_gears_experiments.test:{'MainClassName': 'gears_experiments.test', 'Version': 1, 'Description': 'foo', 'UpgradeData': 'Null', 'Linked': 'true', 'dead': 'false', 'JarFilePath': '/home/meir/work/RedisGears/plugins/jvmplugin/-jars/6876b8b78ccfc2ad764edc7ede590f573bd7260b.jar', 'Registrations': '['0000000000000000000000000000000000000000-1']'} # rg_jvm_stats rg_jvm_mem_stats:["runtimeReport",["heapTotalMemory",1052770304,"heapFreeMemory",1046559936,"heapMaxMemory",16789798912],"memoryMXBeanReport",["heapMemory",{"init":1052770304,"used":4194304,"committed":1052770304,"max":16789798912},"nonHeapMemory",{"init":7667712,"used":9562744,"committed":16318464,"max":-1}],"pools",["CodeHeap 'non-nmethods'",{"init":2555904,"used":1663872,"committed":2555904,"max":7598080},"Metaspace",{"init":0,"used":7922336,"committed":8519680,"max":-1},"CodeHeap 'profiled nmethods'",{"init":2555904,"used":640768,"committed":2555904,"max":122028032},"Compressed Class Space",{"init":0,"used":751464,"committed":917504,"max":1073741824},"G1 Eden Space",{"init":54525952,"used":4194304,"committed":54525952,"max":-1},"G1 Old Gen",{"init":998244352,"used":0,"committed":998244352,"max":16789798912},"G1 Survivor Space",{"init":0,"used":0,"committed":0,"max":-1},"CodeHeap 'non-profiled nmethods'",{"init":2555904,"used":99072,"committed":2555904,"max":122032128}],"totalAllocatedMemory",2131444864,"totalAllocatedMemoryHuman","2032.7042236328125mb"] ``` # Technical Details To allow atomic update of the code a new mechanism was developed to allow accumulate all the registrations created by a session and apply them in a single batch only when we know for sure we are not going to failed. New mechanism implemented under `exeuction_plane.c` and called `SessionRegistrationCtx`. The mechanism provides the following methods: * RG_AddRegistrationToUnregister - add a registration id to unregistered when applying the batch. * RG_AddSessionToUnlink - add a session to unlink when applying the batch * RG_PrepareForRegister - add a registration to register when applying the batch. The batch execution order: * Unlink sessions * Unregister old registrations * Register new registrations The `SessionRegistrationCtx` is also distributed to all the shard in the cluster and apply atomically on each shard. In addition, a new reader callback, `KeysReader_VerifyRegisrter`, was added that allows to verify its OK to register a give registration. If the reader return OK it means that the register the registration on the current batch is going to successes. * do not publish jvmplugin on pull requests (#701) * do not publish jvmplugin on pull requests * change jvm publish to separate job * syntax * straight gears * makefile * changes to limit to branch Co-authored-by: Meir Shpilraien (Spielrein) <meir@redis.com> * Docs copyedit suggestions (#694) * Copyedits for commands_hook.md * Fixed navigation title typos * Copyedits for miss_event.md * Copyedits for async_await_advance_topics.md * More copyedits for async_await_advance_topics.md * More copyedits for miss_event.md * More copyedits for commands_hook.md * Applied review suggestions Co-authored-by: Meir Shpilraien (Spielrein) <meir@redis.com> Co-authored-by: Chayim I. Kirshen <c@kirshen.com> Co-authored-by: Chayim <chayim@users.noreply.github.com> * set version to 1.2.2 * set gears_runtime version to 1.0.0 * added rg.jdumpsessions to ramp Co-authored-by: Leibale Eidelman <leibale1998@gmail.com> Co-authored-by: Chayim Kirshen <c@kirshen.com> Co-authored-by: alonre24 <alonreshef24@gmail.com> Co-authored-by: Rafi Einstein <raffapen@outlook.com> Co-authored-by: Chayim <chayim@users.noreply.github.com> Co-authored-by: Guy Korland <gkorland@gmail.com> Co-authored-by: Danni Moiseyev <isotop@gmail.com> Co-authored-by: Simon Prickett <simon@redislabs.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ofir Moskovich <ofir.moskovich@redislabs.com> Co-authored-by: OfirMos <82366525+OfirMos@users.noreply.github.com> Co-authored-by: João Acabado <jracabado@users.noreply.github.com> Co-authored-by: Chris Mague <maguec@users.noreply.github.com> Co-authored-by: dengliming <liming.d.pro@gmail.com> Co-authored-by: Rachel Elledge <86307637+rrelledge@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
So async executions time will be reported to slowlog and latency monitor.
The text was updated successfully, but these errors were encountered: