Skip to content

Commit 26d5c79

Browse files
committed
Added RIOT
1 parent c2d6b79 commit 26d5c79

File tree

6 files changed

+881
-1
lines changed

6 files changed

+881
-1
lines changed

docs/tools/index-tools.mdx

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
id: index-tools
2+
title: Tools
3+
hide_title: true
4+
sidebar_label: Tools
5+
hide_table_of_contents: true
6+
slug: /tools/
7+
custom_edit_url:
8+
---
9+
import Languages from '@theme/Languages';
10+
11+
<Languages />

docs/tools/riot/index-riot.mdx

Lines changed: 329 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,329 @@
1+
---
2+
id: index-riot
3+
title: RIOT
4+
sidebar_label: RIOT
5+
slug: /riot
6+
---
7+
8+
import Tabs from '@theme/Tabs';
9+
import TabItem from '@theme/TabItem';
10+
import useBaseUrl from '@docusaurus/useBaseUrl';
11+
import RedisCard from '@site/src/theme/RedisCard';
12+
13+
14+
Redis Input/Output Tools (RIOT) is a set of import/export command line utilities for Redis:
15+
16+
- RIOT DB: migrate from an RDBMS to Redis, RediSearch, RedisJSON, ...
17+
- RIOT File: bulk import/export data from/to files.
18+
- RIOT Gen: generate sample Redis datasets for new feature development and proof of concept.
19+
- RIOT Redis: live replication from any Redis database (including AWS Elasticache) to another Redis database.
20+
- RIOT Stream: import/export messages from/to Kafka topics.
21+
22+
23+
24+
![my image](riottool.svg)
25+
26+
27+
28+
29+
<Tabs
30+
defaultValue="RIOT Redis"
31+
values={[
32+
{label: 'RIOT Redis', value: 'RIOT Redis'},
33+
{label: 'RIOT DB', value: 'RIOT DB'},
34+
]}>
35+
<TabItem value="RIOT Redis">
36+
37+
38+
Most database migration tools available today are offline in nature. Migrating data from AWS ElastiCache to Redis Enterprise Cloud for example means backing up your Elasticache data to an AWS S3 bucket and importing it into Redis Enterprise Cloud using its UI.This implies some downtime and might result in data loss.
39+
Other available techniques include creating point-in-time snapshots of the source Redis server & applying the changes to the destination servers to keep both servers in sync.
40+
It might sound like a good approach but can be challenging when you have to maintain dozens of scripts to implement the migration strategy.
41+
42+
RIOT Redis is a migration tool that allows for seamless live replication between two Redis databases.
43+
44+
### 1. Getting Started
45+
46+
47+
Download the [latest release](https://github.com/redis-developer/riot/releases/latest) and unzip the archive.
48+
49+
Launch the `bin/riot-redis` script and follow the usage information provided.
50+
51+
### 2. Build and Run
52+
53+
```
54+
git clone https://github.com/redis-developer/riot.git
55+
cd riot/riot-redis
56+
./riot-redis
57+
```
58+
59+
### 3. Install via Homebrew (macOS)
60+
61+
```
62+
brew install jruaux/tap/riot-redis`
63+
```
64+
65+
### Usage
66+
67+
```
68+
❯ riot-redis
69+
Usage: {app} [OPTIONS] [COMMAND]
70+
--help Show this help message and exit.
71+
-V, --version Print version information and exit.
72+
-q, --quiet Log errors only
73+
-d, --debug Log in debug mode (includes normal stacktrace)
74+
-i, --info Set log level to info
75+
```
76+
77+
You can use --help on any subcommand:
78+
79+
```
80+
❯ riot-redis --help
81+
82+
❯ riot-redis import --help
83+
84+
❯ riot-redis import .. hset --help
85+
```
86+
87+
Redis connection options are the same as redis-cli:
88+
89+
```
90+
-h, --hostname=<host> Server hostname (default: 127.0.0.1)
91+
-p, --port=<port> Server port (default: 6379)
92+
-s, --socket=<socket> Server socket (overrides hostname and port)
93+
--user=<username> Used to send ACL style 'AUTH username pass'. Needs password.
94+
-a, --pass[=<password>] Password to use when connecting to the server
95+
-u, --uri=<uri> Server URI
96+
-o, --timeout=<sec> Redis command timeout (default: 60)
97+
-n, --db=<int> Database number (default: 0)
98+
-c, --cluster Enable cluster mode
99+
-t, --tls Establish a secure TLS connection
100+
-l, --latency Show latency metrics
101+
-m, --pool=<int> Max pool connections (default: 8)
102+
```
103+
Redis URI syntax is described here.
104+
105+
### 4. Example
106+
107+
Here is an example of a live replication from a source Redis running on localhost and port 6379, to a target Redis running on localhost and port 6380:
108+
109+
```
110+
❯ riot-redis -h source -p 6379 replicate --idle-timeout 500 -h target -p 6380 --live
111+
```
112+
113+
### 5. Verification
114+
115+
Once replication is complete RIOT Redis will perform a verification step to compare values and TTLs between source and target databases. The output looks like this:
116+
117+
```
118+
OK:1000 V:0 >:0 <:0 T:0
119+
```
120+
121+
- OK: # identical values
122+
123+
- V: # mismatched values
124+
125+
- >: # keys only present in source database
126+
127+
- <: # keys only present in target database
128+
129+
- T: # keys with TTL difference greater than tolerance
130+
131+
132+
### 6. Architecture
133+
134+
RIOT Redis implements client-side replication using a producer/consumer approach:
135+
136+
- the producer is connected to the source Redis (e.g. ElastiCache) and iterates over keys to read their corresponding values
137+
138+
- the consumer is connected to the target Redis (e.g. Redis Enterprise Cloud) and writes the key/value tuples previously created
139+
140+
1. Key reader: initiates a SCAN and optionally calls SUBSCRIBE to listen for keyspace notifications (live replication).
141+
2. Value reader: takes the keys and calls DUMP and TTL.
142+
3. Key/Value writer: takes key/value/ttl tuples and calls RESTORE and EXPIRE.
143+
144+
Note: Live replication makes use of keyspace notifications. Make sure the source Redis database has keyspace notifications enabled using notify-keyspace-events = KA in redis.conf or via CONFIG SET.
145+
146+
Note: The live replication mechanism does not guarantee data consistency. Redis sends keyspace notifications over pub/sub which does not provide guaranteed delivery. It is possible that RIOT Redis can miss some notifications in case of network failures for example.
147+
148+
149+
150+
</TabItem>
151+
152+
<TabItem value="RIOT DB">
153+
154+
155+
RIOT DB lets you import/export data from relational databases.
156+
157+
### 1. Getting Started
158+
159+
Download the [latest release](https://github.com/redis-developer/riot/releases/latest) and unzip the archive.
160+
161+
Launch the bin/riot-db script and follow the usage information provided.
162+
163+
### 2. Build and Run
164+
165+
```
166+
❯ git clone https://github.com/redis-developer/riot.git
167+
❯ cd riot/riot-db
168+
❯ ./riot-db
169+
```
170+
171+
### 3. Install via Homebrew (macOS)
172+
173+
```
174+
brew install jruaux/tap/riot-db
175+
```
176+
177+
### 4. Usage
178+
179+
```
180+
❯ riot-db
181+
Usage: riot-db [OPTIONS] [COMMAND]
182+
--help Show this help message and exit.
183+
-V, --version Print version information and exit.
184+
-q, --quiet Log errors only
185+
-d, --debug Log in debug mode (includes normal stacktrace)
186+
-i, --info Set log level to info
187+
```
188+
You can use --help on any subcommand:
189+
190+
```
191+
❯ riot-db --help
192+
❯ riot-db import --help
193+
❯ riot-db import … hset --help
194+
```
195+
196+
Redis connection options are the same as redis-cli:
197+
198+
```
199+
-h, --hostname=<host> Server hostname (default: 127.0.0.1)
200+
-p, --port=<port> Server port (default: 6379)
201+
-s, --socket=<socket> Server socket (overrides hostname and port)
202+
--user=<username> Used to send ACL style 'AUTH username pass'. Needs password.
203+
-a, --pass[=<password>] Password to use when connecting to the server
204+
-u, --uri=<uri> Server URI
205+
-o, --timeout=<sec> Redis command timeout (default: 60)
206+
-n, --db=<int> Database number (default: 0)
207+
-c, --cluster Enable cluster mode
208+
-t, --tls Establish a secure TLS connection
209+
-l, --latency Show latency metrics
210+
-m, --pool=<int> Max pool connections (default: 8)
211+
212+
```
213+
### 5. Drivers
214+
215+
RIOT DB includes drivers for the most common RDBMSs:
216+
217+
#### Oracle
218+
219+
```
220+
jdbc:oracle:thin:@myhost:1521:orcl
221+
```
222+
223+
#### IBM Db2
224+
225+
```
226+
jdbc:db2://host:port/database
227+
```
228+
229+
#### MS SQL Server
230+
231+
```
232+
jdbc:sqlserver://[serverName[\instanceName][:portNumber]][;property=value[;property=value]]
233+
```
234+
235+
#### MySQL
236+
237+
```
238+
jdbc:mysql://[host]:[port][/database][?properties]
239+
```
240+
241+
#### PostgreSQL
242+
243+
```
244+
jdbc:postgresql://host:port/database
245+
```
246+
247+
#### SQLite
248+
249+
```
250+
jdbc:sqlite:sqlite_database_file_path
251+
```
252+
253+
For non-included databases you must install the corresponding JDBC driver under the lib directory and modify the RIOT DB CLASSPATH:
254+
255+
```
256+
*nix: bin/riot-db → CLASSPATH=$APP_HOME/lib/myjdbc.jar:$APP_HOME/lib/…
257+
Windows: bin{app}.bat → set CLASSPATH=%APP_HOME%\lib\myjdbc.jar;%APP_HOME%\lib\…
258+
```
259+
260+
### 6. Import
261+
262+
Use the import command to import the result set of a SQL statement.
263+
264+
#### Import from PostgreSQL
265+
266+
```
267+
❯ riot-db -h localhost -p 6379 import "SELECT * FROM orders" --url jdbc:postgresql://host:port/database --username appuser --password passwd hset --keyspace order --keys order_id
268+
```
269+
270+
You can specify one or many Redis commands as targets of the import:
271+
272+
#### Import into hashes
273+
274+
```
275+
❯ riot-db import .. set --keyspace blah --keys id
276+
```
277+
278+
#### Import into hashes and set TTL on the key
279+
280+
```
281+
❯ riot-db import .. hset --keyspace blah --keys id expire --keyspace blah --keys id
282+
```
283+
284+
#### Import into hashes and set TTL and add to a set named myset
285+
286+
```
287+
❯ riot-db import .. hset --keyspace blah --keys id expire --keyspace blah --keys id sadd --keyspace myset --members id
288+
```
289+
290+
### 7. Export
291+
292+
#### Export to PostgreSQL
293+
294+
```
295+
❯ riot-db export "INSERT INTO mytable (id, field1, field2) VALUES (CAST(:id AS SMALLINT), :field1, :field2)" --url jdbc:postgresql://host:port/database --username appuser --password passwd --scan-match "hash:*" --key-regex "hash:(?<id>.*)"
296+
```
297+
298+
#### Import from PostgreSQL to JSON strings
299+
300+
```
301+
❯ riot-db -h localhost -p 6379 import "SELECT * FROM orders" --url jdbc:postgresql://host:port/database --username appuser --password passwd set --keyspace order --keys order_id
302+
```
303+
This will produce Redis strings that look like this:
304+
305+
```
306+
{
307+
"order_id": 10248,
308+
"customer_id": "VINET",
309+
"employee_id": 5,
310+
"order_date": "1996-07-04",
311+
"required_date": "1996-08-01",
312+
"shipped_date": "1996-07-16",
313+
"ship_via": 3,
314+
"freight": 32.38,
315+
"ship_name": "Vins et alcools Chevalier",
316+
"ship_address": "59 rue de l'Abbaye",
317+
"ship_city": "Reims",
318+
"ship_postal_code": "51100",
319+
"ship_country": "France"
320+
}
321+
```
322+
323+
324+
325+
326+
</TabItem>
327+
</Tabs>
328+
329+

0 commit comments

Comments
 (0)