Storage

Peter Stuifzand edited this page Jul 7, 2018 · 4 revisions

What data is contained in Redis?

These commands don't contain all data. Channels and Feeds are loaded from backend.json. Redis contains: items, channel sort order and items in channels.

Items in channels

Items in channels are add to Sorted Sets with the published date as the score and the value is the itemKey. The key matches "zchannel:<channel-key>:posts" where <channel-key> is the channel uid.

Items are removed from these channels when the item is read.

Items

Items are Hashes and contain the following keys: ID, Published, Read, Data. Only Data is used. The other values could be used to reconstruct the channel.

Channel sort order

Every channel has a key "channel_sortorder_"+uid and value 1 - 99999. When channels are retrieved they are sorted with this number. At the moment there is no way to adjust this value through the API.

Cleanup of old items (possible extension)

Currently there is no way in which old items are getting removed. How would this be possible in this scheme?

Currently all items are added to a set of read items. It should be possible to remove all items from there. Maybe some range of items (last 7 days) should be kept.

Multiple users (possible extension)

At the moment it's not possible to use the same Redis server for multiple users. How could this be implemented?

Channels are user specific. Every user has their own set of channels. (user, channel)

Read status is user specific. Every user hash their own set of read status according to the relation (item, channel, read).

Items are global. Removing an item should only happen when no user has access to the item anymore.

Redis commands used in Ekster storage backend

// Load
SETNX "channel_sortorder_notifications", 1

DEL "channels"

// for uid in channels

    SADD "channels", uid
    SETX "channel_sortorder_"+uid, 99999


// ChannelsGetList
SORT "channels" BY "channel_sortorder_*", ASC

// ChannelsDelete
SREM "channels", uid
DEL "channel_sortorder_"+uid

// TimelineGet
channelKey = "zchannel:%s:posts", channel
items = ZRANGEBYSCORE channelKey afterScore beforeScore LIMIT 0 20 WITHSCORE

    // for item in items
    HGET itemId, "Data"

// MarkRead
channelReadKey = "channel:%s:read", channel
SADD channelReadKey, itemIds
channelKey = "zchannel:%s:posts"
ZREM channelKey, itemIds

// channelAddItem
channelKey = "channel:%s:posts"
zchannelKey = "zchannel:%s:posts"

item = {
	ID:        item.ID,
	Published: item.Published,
	Read:      item.Read,
	Data:      data,
}

itemKey = "item:%s", item.ID
HMSET itemKey, item

readChannelKey := "channel:%s:read", channel
isRead = SISMEMBER readChannelKey, itemKey
if isRead return

SADD channelKey, itemKey
ZADD zchannelKey, (published as unixtime), itemKey

unread = ZCARD zchannelKey
Clone this wiki locally
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.