v0.1.3

What's Changed Since v0.1.2

Fixes

• Hash reads returned wrong data after column pruning. The hash explode-vs-wide read mode was decided from the (possibly pruned) required schema, so queries that prune the field column (e.g. SELECT key, value, SELECT count(*)) silently switched to wide-hash mode and returned wrong rows. The mode is now decided once from the full table schema and threaded down to the partition reader.
• Writer abort() now actually discards buffered writes. abort() previously only set a flag while close() always called pipeline.close(), which syncs buffered commands. Writers now use a dedicated (non-pooled) connection so abort() closes the socket and drops unsent commands.
• Jedis connection pools are closed on JVM shutdown. A shutdown hook plus closeAll() prevent pools leaking for the lifetime of the driver/executor.
• EXPIRE is issued once per key instead of once per row. Removes redundant TTL commands (and repeated TTL resets) for set/list/zset/hash-field writes.
• Removed unused RedisRowCodec helpers (hashRow, optionalStringAt) and a now-unused import.

Upgrade Notes

• Strongly recommended if you read Redis hashes in field/value layout and run queries that do not select the field column (for example SELECT key, value or SELECT count(*)). Versions before 0.1.3 could return silently wrong rows after column pruning.
• Writes remain non-transactional. abort() can discard commands buffered but not yet sent; rows already flushed to Redis are not rolled back.
• TTL writes: set/list/zset/hash-field writes now send at most one EXPIRE per Redis key per writer task instead of one per row.

Verification

mvn clean package

Full Changelog: v0.1.2...v0.1.3

v0.1.2

What's Changed Since v0.1.1

Features

• Support read-time key.prefix stripping. Redis keys such as bigdata:biz:uid:777 can now be returned to Spark as 777, while write-time prefix behavior remains unchanged.
• Add RedisKeyCodec for consistent logical key and Redis key conversion.

Documentation

• Split Redis type guides into dedicated docs under docs/types/.
• Add Spark compatibility guidance for Spark 3.3.x, 3.4.x, and 3.5.x with Scala 2.12.
• Add Chinese README at README.zh-CN.md.
• Clarify type-specific option sections in type docs.
• Improve README language switch visibility.

Upgrade Notes

• If you use prefixed Redis keys, configure key.prefix on reads to get logical Spark keys without the prefix.
• Example:
  • keys.pattern='bigdata:biz:uid:*'
  • key.prefix='bigdata:biz:uid:'

Verification

• mvn clean package

v0.1.1

What's Changed

• Add executor-side Jedis connection pooling for Redis reads, writes, scans, and overwrite deletes.
• Add SSL/TLS support with optional truststore configuration.
• Add pipelined Redis writes controlled by write.pipeline.size.
• Simplify the public option set for the early connector API.
• Keep a single timeout option for both Redis connection and socket timeouts.
• Remove redundant aliases such as auth, dbNum, ssl, write.batch.size, and pipeline.size.
• Update README examples and release artifact names for 0.1.1.

Verification

• mvn clean package

v0.1.0

Initial MVP release of spark-redis-connector.

Highlights:

• Spark SQL DataSource V2 connector with USING redis short name.
• Batch read/write support for Redis string, hash, set, list, and sorted set.
• Hash narrow and flattened layouts.
• List explode and array read modes, plus configurable LPUSH/RPUSH writes.
• Shaded all jar for local Spark testing.