|
| 1 | +--- |
| 2 | +id: index-riot |
| 3 | +title: RIOT |
| 4 | +sidebar_label: RIOT |
| 5 | +slug: /riot |
| 6 | +--- |
| 7 | + |
| 8 | +import Tabs from '@theme/Tabs'; |
| 9 | +import TabItem from '@theme/TabItem'; |
| 10 | +import useBaseUrl from '@docusaurus/useBaseUrl'; |
| 11 | +import RedisCard from '@site/src/theme/RedisCard'; |
| 12 | + |
| 13 | + |
| 14 | +Redis Input/Output Tools (RIOT) is a set of import/export command line utilities for Redis: |
| 15 | + |
| 16 | +- RIOT DB: migrate from an RDBMS to Redis, RediSearch, RedisJSON, ... |
| 17 | +- RIOT File: bulk import/export data from/to files. |
| 18 | +- RIOT Gen: generate sample Redis datasets for new feature development and proof of concept. |
| 19 | +- RIOT Redis: live replication from any Redis database (including AWS Elasticache) to another Redis database. |
| 20 | +- RIOT Stream: import/export messages from/to Kafka topics. |
| 21 | + |
| 22 | + |
| 23 | + |
| 24 | + |
| 25 | + |
| 26 | + |
| 27 | + |
| 28 | + |
| 29 | +<Tabs |
| 30 | + defaultValue="RIOT Redis" |
| 31 | + values={[ |
| 32 | + {label: 'RIOT Redis', value: 'RIOT Redis'}, |
| 33 | + {label: 'RIOT DB', value: 'RIOT DB'}, |
| 34 | + ]}> |
| 35 | + <TabItem value="RIOT Redis"> |
| 36 | + |
| 37 | + |
| 38 | +Most database migration tools available today are offline in nature. Migrating data from AWS ElastiCache to Redis Enterprise Cloud for example means backing up your Elasticache data to an AWS S3 bucket and importing it into Redis Enterprise Cloud using its UI.This implies some downtime and might result in data loss. |
| 39 | +Other available techniques include creating point-in-time snapshots of the source Redis server & applying the changes to the destination servers to keep both servers in sync. |
| 40 | +It might sound like a good approach but can be challenging when you have to maintain dozens of scripts to implement the migration strategy. |
| 41 | + |
| 42 | +RIOT Redis is a migration tool that allows for seamless live replication between two Redis databases. |
| 43 | + |
| 44 | +### 1. Getting Started |
| 45 | + |
| 46 | + |
| 47 | +Download the [latest release](https://github.com/redis-developer/riot/releases/latest) and unzip the archive. |
| 48 | + |
| 49 | +Launch the `bin/riot-redis` script and follow the usage information provided. |
| 50 | + |
| 51 | +### 2. Build and Run |
| 52 | + |
| 53 | +``` |
| 54 | +git clone https://github.com/redis-developer/riot.git |
| 55 | +cd riot/riot-redis |
| 56 | +./riot-redis |
| 57 | +``` |
| 58 | + |
| 59 | +### 3. Install via Homebrew (macOS) |
| 60 | + |
| 61 | +``` |
| 62 | +brew install jruaux/tap/riot-redis` |
| 63 | +``` |
| 64 | + |
| 65 | +### Usage |
| 66 | + |
| 67 | +``` |
| 68 | +❯ riot-redis |
| 69 | +Usage: {app} [OPTIONS] [COMMAND] |
| 70 | + --help Show this help message and exit. |
| 71 | + -V, --version Print version information and exit. |
| 72 | + -q, --quiet Log errors only |
| 73 | + -d, --debug Log in debug mode (includes normal stacktrace) |
| 74 | + -i, --info Set log level to info |
| 75 | +``` |
| 76 | + |
| 77 | +You can use --help on any subcommand: |
| 78 | + |
| 79 | +``` |
| 80 | +❯ riot-redis --help |
| 81 | +
|
| 82 | +❯ riot-redis import --help |
| 83 | +
|
| 84 | +❯ riot-redis import .. hset --help |
| 85 | +``` |
| 86 | + |
| 87 | +Redis connection options are the same as redis-cli: |
| 88 | + |
| 89 | +``` |
| 90 | + -h, --hostname=<host> Server hostname (default: 127.0.0.1) |
| 91 | + -p, --port=<port> Server port (default: 6379) |
| 92 | + -s, --socket=<socket> Server socket (overrides hostname and port) |
| 93 | + --user=<username> Used to send ACL style 'AUTH username pass'. Needs password. |
| 94 | + -a, --pass[=<password>] Password to use when connecting to the server |
| 95 | + -u, --uri=<uri> Server URI |
| 96 | + -o, --timeout=<sec> Redis command timeout (default: 60) |
| 97 | + -n, --db=<int> Database number (default: 0) |
| 98 | + -c, --cluster Enable cluster mode |
| 99 | + -t, --tls Establish a secure TLS connection |
| 100 | + -l, --latency Show latency metrics |
| 101 | + -m, --pool=<int> Max pool connections (default: 8) |
| 102 | +``` |
| 103 | +Redis URI syntax is described here. |
| 104 | + |
| 105 | +### 4. Example |
| 106 | + |
| 107 | +Here is an example of a live replication from a source Redis running on localhost and port 6379, to a target Redis running on localhost and port 6380: |
| 108 | + |
| 109 | +``` |
| 110 | +❯ riot-redis -h source -p 6379 replicate --idle-timeout 500 -h target -p 6380 --live |
| 111 | +``` |
| 112 | + |
| 113 | +### 5. Verification |
| 114 | + |
| 115 | +Once replication is complete RIOT Redis will perform a verification step to compare values and TTLs between source and target databases. The output looks like this: |
| 116 | + |
| 117 | +``` |
| 118 | +OK:1000 V:0 >:0 <:0 T:0 |
| 119 | +``` |
| 120 | + |
| 121 | +- OK: # identical values |
| 122 | + |
| 123 | +- V: # mismatched values |
| 124 | + |
| 125 | +- >: # keys only present in source database |
| 126 | +
|
| 127 | +- <: # keys only present in target database |
| 128 | + |
| 129 | +- T: # keys with TTL difference greater than tolerance |
| 130 | + |
| 131 | + |
| 132 | +### 6. Architecture |
| 133 | + |
| 134 | +RIOT Redis implements client-side replication using a producer/consumer approach: |
| 135 | + |
| 136 | +- the producer is connected to the source Redis (e.g. ElastiCache) and iterates over keys to read their corresponding values |
| 137 | + |
| 138 | +- the consumer is connected to the target Redis (e.g. Redis Enterprise Cloud) and writes the key/value tuples previously created |
| 139 | + |
| 140 | +1. Key reader: initiates a SCAN and optionally calls SUBSCRIBE to listen for keyspace notifications (live replication). |
| 141 | +2. Value reader: takes the keys and calls DUMP and TTL. |
| 142 | +3. Key/Value writer: takes key/value/ttl tuples and calls RESTORE and EXPIRE. |
| 143 | + |
| 144 | +Note: Live replication makes use of keyspace notifications. Make sure the source Redis database has keyspace notifications enabled using notify-keyspace-events = KA in redis.conf or via CONFIG SET. |
| 145 | + |
| 146 | +Note: The live replication mechanism does not guarantee data consistency. Redis sends keyspace notifications over pub/sub which does not provide guaranteed delivery. It is possible that RIOT Redis can miss some notifications in case of network failures for example. |
| 147 | + |
| 148 | + |
| 149 | + |
| 150 | +</TabItem> |
| 151 | + |
| 152 | +<TabItem value="RIOT DB"> |
| 153 | + |
| 154 | + |
| 155 | +RIOT DB lets you import/export data from relational databases. |
| 156 | + |
| 157 | +### 1. Getting Started |
| 158 | + |
| 159 | +Download the [latest release](https://github.com/redis-developer/riot/releases/latest) and unzip the archive. |
| 160 | + |
| 161 | +Launch the bin/riot-db script and follow the usage information provided. |
| 162 | + |
| 163 | +### 2. Build and Run |
| 164 | + |
| 165 | +``` |
| 166 | +❯ git clone https://github.com/redis-developer/riot.git |
| 167 | +❯ cd riot/riot-db |
| 168 | +❯ ./riot-db |
| 169 | +``` |
| 170 | + |
| 171 | +### 3. Install via Homebrew (macOS) |
| 172 | + |
| 173 | +``` |
| 174 | +brew install jruaux/tap/riot-db |
| 175 | +``` |
| 176 | + |
| 177 | +### 4. Usage |
| 178 | + |
| 179 | +``` |
| 180 | +❯ riot-db |
| 181 | +Usage: riot-db [OPTIONS] [COMMAND] |
| 182 | + --help Show this help message and exit. |
| 183 | + -V, --version Print version information and exit. |
| 184 | + -q, --quiet Log errors only |
| 185 | + -d, --debug Log in debug mode (includes normal stacktrace) |
| 186 | + -i, --info Set log level to info |
| 187 | +``` |
| 188 | +You can use --help on any subcommand: |
| 189 | + |
| 190 | +``` |
| 191 | +❯ riot-db --help |
| 192 | +❯ riot-db import --help |
| 193 | +❯ riot-db import … hset --help |
| 194 | +``` |
| 195 | + |
| 196 | +Redis connection options are the same as redis-cli: |
| 197 | + |
| 198 | +``` |
| 199 | + -h, --hostname=<host> Server hostname (default: 127.0.0.1) |
| 200 | + -p, --port=<port> Server port (default: 6379) |
| 201 | + -s, --socket=<socket> Server socket (overrides hostname and port) |
| 202 | + --user=<username> Used to send ACL style 'AUTH username pass'. Needs password. |
| 203 | + -a, --pass[=<password>] Password to use when connecting to the server |
| 204 | + -u, --uri=<uri> Server URI |
| 205 | + -o, --timeout=<sec> Redis command timeout (default: 60) |
| 206 | + -n, --db=<int> Database number (default: 0) |
| 207 | + -c, --cluster Enable cluster mode |
| 208 | + -t, --tls Establish a secure TLS connection |
| 209 | + -l, --latency Show latency metrics |
| 210 | + -m, --pool=<int> Max pool connections (default: 8) |
| 211 | +
|
| 212 | +``` |
| 213 | +### 5. Drivers |
| 214 | + |
| 215 | +RIOT DB includes drivers for the most common RDBMSs: |
| 216 | + |
| 217 | +#### Oracle |
| 218 | + |
| 219 | +``` |
| 220 | +jdbc:oracle:thin:@myhost:1521:orcl |
| 221 | +``` |
| 222 | + |
| 223 | +#### IBM Db2 |
| 224 | + |
| 225 | +``` |
| 226 | +jdbc:db2://host:port/database |
| 227 | +``` |
| 228 | + |
| 229 | +#### MS SQL Server |
| 230 | + |
| 231 | +``` |
| 232 | +jdbc:sqlserver://[serverName[\instanceName][:portNumber]][;property=value[;property=value]] |
| 233 | +``` |
| 234 | + |
| 235 | +#### MySQL |
| 236 | + |
| 237 | +``` |
| 238 | +jdbc:mysql://[host]:[port][/database][?properties] |
| 239 | +``` |
| 240 | + |
| 241 | +#### PostgreSQL |
| 242 | + |
| 243 | +``` |
| 244 | +jdbc:postgresql://host:port/database |
| 245 | +``` |
| 246 | + |
| 247 | +#### SQLite |
| 248 | + |
| 249 | +``` |
| 250 | +jdbc:sqlite:sqlite_database_file_path |
| 251 | +``` |
| 252 | + |
| 253 | +For non-included databases you must install the corresponding JDBC driver under the lib directory and modify the RIOT DB CLASSPATH: |
| 254 | + |
| 255 | +``` |
| 256 | +*nix: bin/riot-db → CLASSPATH=$APP_HOME/lib/myjdbc.jar:$APP_HOME/lib/… |
| 257 | +Windows: bin{app}.bat → set CLASSPATH=%APP_HOME%\lib\myjdbc.jar;%APP_HOME%\lib\… |
| 258 | +``` |
| 259 | + |
| 260 | +### 6. Import |
| 261 | + |
| 262 | +Use the import command to import the result set of a SQL statement. |
| 263 | + |
| 264 | +#### Import from PostgreSQL |
| 265 | + |
| 266 | +``` |
| 267 | +❯ riot-db -h localhost -p 6379 import "SELECT * FROM orders" --url jdbc:postgresql://host:port/database --username appuser --password passwd hset --keyspace order --keys order_id |
| 268 | +``` |
| 269 | + |
| 270 | +You can specify one or many Redis commands as targets of the import: |
| 271 | + |
| 272 | +#### Import into hashes |
| 273 | + |
| 274 | +``` |
| 275 | +❯ riot-db import .. set --keyspace blah --keys id |
| 276 | +``` |
| 277 | + |
| 278 | +#### Import into hashes and set TTL on the key |
| 279 | + |
| 280 | +``` |
| 281 | +❯ riot-db import .. hset --keyspace blah --keys id expire --keyspace blah --keys id |
| 282 | +``` |
| 283 | + |
| 284 | +#### Import into hashes and set TTL and add to a set named myset |
| 285 | + |
| 286 | +``` |
| 287 | +❯ riot-db import .. hset --keyspace blah --keys id expire --keyspace blah --keys id sadd --keyspace myset --members id |
| 288 | +``` |
| 289 | + |
| 290 | +### 7. Export |
| 291 | + |
| 292 | +#### Export to PostgreSQL |
| 293 | + |
| 294 | +``` |
| 295 | +❯ riot-db export "INSERT INTO mytable (id, field1, field2) VALUES (CAST(:id AS SMALLINT), :field1, :field2)" --url jdbc:postgresql://host:port/database --username appuser --password passwd --scan-match "hash:*" --key-regex "hash:(?<id>.*)" |
| 296 | +``` |
| 297 | + |
| 298 | +#### Import from PostgreSQL to JSON strings |
| 299 | + |
| 300 | +``` |
| 301 | +❯ riot-db -h localhost -p 6379 import "SELECT * FROM orders" --url jdbc:postgresql://host:port/database --username appuser --password passwd set --keyspace order --keys order_id |
| 302 | +``` |
| 303 | +This will produce Redis strings that look like this: |
| 304 | + |
| 305 | +``` |
| 306 | +{ |
| 307 | + "order_id": 10248, |
| 308 | + "customer_id": "VINET", |
| 309 | + "employee_id": 5, |
| 310 | + "order_date": "1996-07-04", |
| 311 | + "required_date": "1996-08-01", |
| 312 | + "shipped_date": "1996-07-16", |
| 313 | + "ship_via": 3, |
| 314 | + "freight": 32.38, |
| 315 | + "ship_name": "Vins et alcools Chevalier", |
| 316 | + "ship_address": "59 rue de l'Abbaye", |
| 317 | + "ship_city": "Reims", |
| 318 | + "ship_postal_code": "51100", |
| 319 | + "ship_country": "France" |
| 320 | +} |
| 321 | +``` |
| 322 | + |
| 323 | + |
| 324 | + |
| 325 | + |
| 326 | +</TabItem> |
| 327 | +</Tabs> |
| 328 | + |
| 329 | + |
0 commit comments