Adds Redis Pub/Sub integration #80

MCHSL · 2021-12-22T13:55:14Z

Adds the ability to connect to Redis, subscribe to messages and publish them as well. See here for example usage.

The API, prefixed with rustg_redis_:

connect(addr) - Connects to a Redis instance using the given address, for example redis://127.0.0.1/. Returns an empty string on success, returns the error otherwise.
disconnect() - Closes the connection to Redis and stops the thread managing it. Call this before restarting or attempting to reconnect after an error.
subscribe(channel) - Subscribes to a given channel and starts receiving messages from it.
get_messages() - Returns all received messages as a JSON string, in the format of
{"channel_1": ["msg1", "msg2", "msg3", ...], "channel_2": ["msg1", "msg2", "msg3", ...]}.
Also includes errors, which appear on the channel "RUSTG_REDIS_ERROR_CHANNEL".
publish(channel, message) - Publishes a message on the given channel.

Remember to check the error channel every time you call get_messages(). If any occur, you need to call disconnect(), then connect() again, then resubscribe to desired channels.

AffectedArc07 · 2021-12-22T14:09:09Z

PR is WIP while I stability test it on para prod

AnturK · 2021-12-22T14:28:17Z

Shouldn't these channels be bounded, then i guess if you subscribe to anything you'll be checking with some regularity.

AffectedArc07 · 2021-12-22T14:28:52Z

Shouldn't these channels be bounded, then i guess if you subscribe to anything you'll be checking with some regularity.

Bounded? In what way?

AffectedArc07 · 2021-12-22T14:34:26Z

Cargo.toml

@@ -41,8 +41,9 @@ mysql = { version = "20.0", optional = true }
 dashmap = { version = "4.0", optional = true }
 zip = { version = "0.5.8", optional = true }
 rand = {version = "0.8", optional = true}
-dmsort = {version = "1.0.0", optional = true}
+dmsort = {version = "1.0.0", optional = true }


Also you might want to bump the version of the package itself
version = "0.5.0"

I guess with semver this would be version = "0.6.0"?

why would you include the patch version

why would you include the patch version

I assumed that was standard with semver. RUSTG has always been 0.x.0

MCHSL · 2021-12-22T14:47:38Z

Bounded? In what way?

Bounded in the sense of only allowing up to some number of messages waiting in queue, right now if you don't consume them, they can pile up forever until you run out of memory. I s'pose I can cap it to 1000 messages or something

optimumtact · 2021-12-22T22:12:00Z

great, i386 pkgs are broken again

AffectedArc07 · 2021-12-22T22:25:59Z

great, i386 pkgs are broken again

ParadiseSS13/rust-g#1

Hacky but fixes it

optimumtact · 2021-12-23T03:32:30Z

surely that's needed for building the mariadb client lib

AffectedArc07 · 2021-12-23T10:29:43Z

surely that's needed for building the mariadb client lib

It used to be needed for OpenSSL but isn't anymore. CI passed without pkg-config even when building with all features.

AffectedArc07 · 2021-12-23T16:31:36Z

Seems to be stable on para.

willox · 2021-12-23T18:00:30Z

pls merge/rebase with master

MCHSL · 2021-12-23T18:52:12Z

Changed it so if the receive queue is full, any further received messages are immediately dropped. Otherwise, it'd prevent sending, subscribing, and stopping from working, and could even deadlock if you attempted to send more than 1k messages while it was blocked.

MCHSL · 2021-12-23T23:35:22Z

Can now handle calling connect several times and forgetting to call disconnect before connecting again

Cargo.toml

README.md

Mothblocks · 2021-12-24T22:31:15Z

src/redis_pubsub.rs

+    out: flume::Sender<PubSubResponse>,
+) {
+    if let Err(e) = handle_redis_inner(client, &control, &out) {
+        let _ = out.try_send(PubSubResponse::Disconnected(e.to_string()));


Why are we silencing potential errors here? This is spooky.

This could fail if the receiving end was dropped, in which case there's no need (or way) to communicate back the error, or if the channel was full, in which case the DM side was probably FUBAR. I suppose it doesn't hurt to make it block until there's space or if the receiver is dropped.

If this blocks, how is the queue ever going to clear if DM cant execute anything to clear the queue due to being blocked by this?

This runs on the redis thread side, it won't block DM

src/redis_pubsub.rs

AffectedArc07 · 2021-12-25T00:09:08Z

Once changes are addressed please do not immediately merge, as I need to stability test it on prod.

Mothblocks · 2021-12-25T01:02:30Z

Draft then plz

MCHSL · 2021-12-28T13:47:03Z

subscribe and publish now also return errors, they can stem either from failing to connect or being unable to keep up with incoming messages and letting buffers fill up

AffectedArc07 · 2021-12-29T20:20:46Z

TM'd on paradise prod for about a day straight with no crashes and decent levels of activity (relaying all msay and asay messages to and from discord)

* Updates the byond testing version to tgstation ver (tgstation#66) * runs cargo fmt (tgstation#63) * Sets the debug and test linux build to use all features (tgstation#67) Problem: PRs and all features aren't properly running created unit tests. Nor are they actually being fully compiled (only checked) on Linux. Solution: Only have the release builds that are distributed as artifacts run with the default features. This way, tests can cover all features. * Base64 encoding (tgstation#65) * Adds base64 encoding * Adds tests * Commented out import * Fixes the naming of variables * ...of course it was a typo * Update Cargo.toml Co-authored-by: William Wallace <me@wiox.me> * Clippy tweaks (tgstation#62) * this `else { if .. }` block can be collapsed * question mark operator is useless here * this call to `from_str_radix` can be replaced with a call to `str::parse` * equality checks against true are unnecessary * Revert "question mark operator is useless here" This reverts commit 01272c5. * Cargo update (tgstation#64) * updates cargo dependencies versions to latest * Fixes perlin noise generation to use an inclusive range * Update src/cellularnoise.rs Co-authored-by: William Wallace <me@wiox.me> * Updates cargo.lock Co-authored-by: William Wallace <me@wiox.me> * Basic version getter (tgstation#68) * Bump Version sto v0.4.8 * Setup cross compiling using the cross project (tgstation#70) * Make the url feature default (tgstation#71) Co-authored-by: Jared-Fogle <35135081+Jared-Fogle@users.noreply.github.com> * 0.4.9 * Runs cargo update (tgstation#72) most importantly git2, which made it error when trying to build on linux with the latest rust version * Stringify url_encode and decode input before sending it to Rust (tgstation#73) Co-authored-by: Jordan Brown <Cyberboss@users.noreply.github.com> * 0.4.10 * Add worley noise optional feature (tgstation#74) * worley_noise * a simple fix, forgot to ad dmsort as a necessary package * replaces .unwrap() with ? * forgot to save :cryingintoheavens: * IM SORRY I CANT USE ? IN CLOSURES * Completely reworks how the code works, massively optimizing it. Generation lost a tiny bit of it's fidelity but oh well, looks fantastic rn * one last change * tfw errors * makes the noise generate pretties noise * cargo fmt * Add TOML feature (tgstation#75) Adds rustg_read_toml_file, which takes a filepath and spits out the output after decoding (a list, probably). Uses the rust-toml lib * 0.5.0 Cargo updates * Adds TOTP generator to the hash feature (tgstation#76) * Adds TOTP generator to the hash feature * Updated TOTP based on requested changes, added a function that allow specification of a tolerance, also updated the hash.dm file to reflect the new functions * Remove debug print, convert offset output to be JSON, added a test * Added comments * Added some error handling * Improves error handling again * cargo fmt hash.rs * separate mod block for hash tests * Functions for more precise time measurement (tgstation#77) * fix linux ci by removing pkg-config requirement This isnt needed * Aho-Corasick string replacements to help clean up replaceText spam (tgstation#82) * Cleanups and additions for the Aho-Corasick replacement stuff (tgstation#83) * Fix clippy lints, Rustfmt, put both on CI, and update Cargo.lock (tgstation#85) Co-authored-by: Mothblocks <35135081+Jared-Fogle@users.noreply.github.com> * Adds Redis Pub/Sub integration (tgstation#80) dds the ability to connect to Redis, subscribe to messages and publish them as well. See here for example usage. The API, prefixed with rustg_redis_: connect(addr) - Connects to a Redis instance using the given address, for example redis://127.0.0.1/. Returns an empty string on success, returns the error otherwise. disconnect() - Closes the connection to Redis and stops the thread managing it. Call this before restarting or attempting to reconnect after an error. subscribe(channel) - Subscribes to a given channel and starts receiving messages from it. get_messages() - Returns all received messages as a JSON string, in the format of {"channel_1": ["msg1", "msg2", "msg3", ...], "channel_2": ["msg1", "msg2", "msg3", ...]}. Also includes errors, which appear on the channel "RUSTG_REDIS_ERROR_CHANNEL". publish(channel, message) - Publishes a message on the given channel. Remember to check the error channel every time you call get_messages(). If any occur, you need to call disconnect(), then connect() again, then resubscribe to desired channels. * fixes warning in redis (tgstation#88) * Fix clippy CI, format byond_fn (tgstation#89) * Adds alphabetical order tests for README.md, lib.rs and Cargo.toml (tgstation#84) Adds alphabetical order tests for README.md, lib.rs and Cargo.toml tgstation#84 Also renamed "non-default features" to "additional features" in Cargo.toml for consistency with readme updated the cargo.lock and ran rustfmt * Bump to 0.6.0 (tgstation#90) * Fixes redis dmsrc (tgstation#91) * Cleaner and saner http request parsing. (tgstation#93) * Add functions for seeking lines and getting line counts in functions (tgstation#95) dds rustg_file_get_line_count for getting the line count of a given filename, and rustg_file_seek_line for getting a specific line. These are useful as to not load the entire file into memory in DM, which is useful for very large files like dictionaries. * 0.7.0 (tgstation#96) * Adds a new type of noise: Discrete Batched Perlin-like Noise (DBP) (tgstation#99) * Updates WorleyNoise to be multi-threaded and faster (tgstation#102) Another noise algorithm updated in my streak, this time it's worleynoise, since i didn't know jack shit and i heavily abused Rc<> to acomplish my goals it was really shit. Now it is multi-threaded, blazing fast and really sexy lookin. NOTE: this update changes ABI of worley noise slightly. * Updates CellularNoise to be faster and multi-threaded (tgstation#101) Hello, I guess i'm on a streak. I reimplemented CAnoise to be multi-threaded and much faster as i removed a considerable amount of branches. No change to behaviour or ABI noted, it's just faster now. * toml2json now returns errors to byond (tgstation#98) Co-authored-by: ZeWaka <zewakagamer@gmail.com> * Updates cargo packages (tgstation#104) * 0.8.0 * Tiny README update * Small improvements for before 1.0 (tgstation#105) * 1.0.0 (tgstation#106) * Pallette Encoding Image Fix (tgstation#107) * Uploads rust_g.dm as an Actions Artifact (tgstation#108) * 1.0.1 * add note to README about not using the .so * Returns the TOML library to previous functionality (tgstation#112) * 1.0.2 * Clippy - unneeded returns (tgstation#114) * Add an a* pathfinder (tgstation#113) Returns the shortest path in a static node map. That is made for TGMC wich uses manually placed nodes for pathfinding. Benchmark : image That's the average number of path computed in one second, out of 30 runs with random nodes. Size of the node map is roughly 800 On average 32 times faster than tgmc a* implementation. Another possible comparison is with TG's JPS system, and according to the benchmark done here it's 7000 times faster (tgstation/tgstation#56780). Not exactly the same use cases though * Add rustg_toml_encode (tgstation#116) Add rustg_toml_encode * 1.1.0 * Fixes cellular noise not working for diff height and width values (tgstation#117) * Update README.md update readme with `pathfinder` feature and rust-analyzer note, re: tgstation#115 * Fix the `pathfinder` feature's loc in the README * Package debug symbols with rust-g (tgstation#119) * 515 Support (tgstation#121) * add a unix_timestamp extern (tgstation#122) Co-authored-by: Mothblocks <35135081+Mothblocks@users.noreply.github.com> * 1.2.0 * Remove useless features from the image crate (tgstation#128) * lmao we don't need jpeg dependencies for image * update `noise` to remove the rest of the junk deps * small clippy fixes (cherry picked from commit 65cf48a) * Update rust.yml * adds all-features flag to linux and windows build * reverts actions to tg's current * Update rust.yml * Update rust.yml * Update rust.yml * Update rust.yml --------- Co-authored-by: ZeWaka <zewakagamer@gmail.com> Co-authored-by: William Wallace <me@wiox.me> Co-authored-by: oranges <email@oranges.net.nz> Co-authored-by: Mothblocks <35135081+Mothblocks@users.noreply.github.com> Co-authored-by: Jared-Fogle <35135081+Jared-Fogle@users.noreply.github.com> Co-authored-by: Mark Suckerberg <29362068+MarkSuckerberg@users.noreply.github.com> Co-authored-by: Gamer025 <33846895+Gamer025@users.noreply.github.com> Co-authored-by: Jordan Brown <Cyberboss@users.noreply.github.com> Co-authored-by: EdgeLordExe <42111655+EdgeLordExe@users.noreply.github.com> Co-authored-by: adamsong <adamsong@users.noreply.github.com> Co-authored-by: pali <6pali6@gmail.com> Co-authored-by: AffectedArc07 <25063394+AffectedArc07@users.noreply.github.com> Co-authored-by: vuonojenmustaturska <naksu@youzen.ext.b2.fi> Co-authored-by: MCHSL <56649176+MCHSL@users.noreply.github.com> Co-authored-by: AnturK <AnturK@users.noreply.github.com> Co-authored-by: tralezab <40974010+tralezab@users.noreply.github.com> Co-authored-by: san7890 <the@san7890.com> Co-authored-by: BraveMole <bsouchu@gmail.com> Co-authored-by: Zephyr <12817816+ZephyrTFA@users.noreply.github.com>

MCHSL added 2 commits December 22, 2021 14:29

add redis pubsub library

279c90d

fix typo

d5d67ff

AffectedArc07 reviewed Dec 22, 2021

View reviewed changes

bound channels to 1000 messages, bump version to 0.6.0

d628fae

update readme

bdaaa97

MCHSL added 2 commits December 23, 2021 19:47

discard overflowing messages

a360fcd

Merge branch 'master' of https://github.com/tgstation/rust-g into redis

b9ab17d

MCHSL marked this pull request as ready for review December 23, 2021 19:27

increase robustness

17c6251

actually attempt connecting in the connect() method

1fc0a22

Mothblocks requested changes Dec 24, 2021

View reviewed changes

alphabetize, add questionable error handling

be5d478

MCHSL marked this pull request as draft December 28, 2021 13:44

MCHSL marked this pull request as ready for review December 29, 2021 20:21

optimumtact merged commit 8c7d298 into tgstation:master Dec 29, 2021

MCHSL deleted the redis branch December 29, 2021 22:41

AffectedArc07 mentioned this pull request Jan 2, 2022

Bump to 0.6.0 #90

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds Redis Pub/Sub integration #80

Adds Redis Pub/Sub integration #80

MCHSL commented Dec 22, 2021

AffectedArc07 commented Dec 22, 2021

AnturK commented Dec 22, 2021

AffectedArc07 commented Dec 22, 2021

AffectedArc07 Dec 22, 2021

ZeWaka Dec 22, 2021

AffectedArc07 Dec 23, 2021

MCHSL commented Dec 22, 2021

optimumtact commented Dec 22, 2021

AffectedArc07 commented Dec 22, 2021

optimumtact commented Dec 23, 2021

AffectedArc07 commented Dec 23, 2021

AffectedArc07 commented Dec 23, 2021

willox commented Dec 23, 2021

MCHSL commented Dec 23, 2021

MCHSL commented Dec 23, 2021

Mothblocks Dec 24, 2021

MCHSL Dec 28, 2021

AffectedArc07 Dec 28, 2021

MCHSL Dec 28, 2021

AffectedArc07 commented Dec 25, 2021

Mothblocks commented Dec 25, 2021

MCHSL commented Dec 28, 2021

AffectedArc07 commented Dec 29, 2021

Adds Redis Pub/Sub integration #80

Adds Redis Pub/Sub integration #80

Conversation

MCHSL commented Dec 22, 2021

AffectedArc07 commented Dec 22, 2021

AnturK commented Dec 22, 2021

AffectedArc07 commented Dec 22, 2021

AffectedArc07 Dec 22, 2021

Choose a reason for hiding this comment

ZeWaka Dec 22, 2021

Choose a reason for hiding this comment

AffectedArc07 Dec 23, 2021

Choose a reason for hiding this comment

MCHSL commented Dec 22, 2021

optimumtact commented Dec 22, 2021

AffectedArc07 commented Dec 22, 2021

optimumtact commented Dec 23, 2021

AffectedArc07 commented Dec 23, 2021

AffectedArc07 commented Dec 23, 2021

willox commented Dec 23, 2021

MCHSL commented Dec 23, 2021

MCHSL commented Dec 23, 2021

Mothblocks Dec 24, 2021

Choose a reason for hiding this comment

MCHSL Dec 28, 2021

Choose a reason for hiding this comment

AffectedArc07 Dec 28, 2021

Choose a reason for hiding this comment

MCHSL Dec 28, 2021

Choose a reason for hiding this comment

AffectedArc07 commented Dec 25, 2021

Mothblocks commented Dec 25, 2021

MCHSL commented Dec 28, 2021

AffectedArc07 commented Dec 29, 2021