ft: ZENKO-147 Use Redis keys instead of hash #286

bennettbuchanan · 2018-05-10T21:15:52Z

We want to be able to expire entries from failed CRR after a configurable amount of time (default is 24 hours). Unfortunately Redis does not support expiry of hash data structure fields, so this PR moves the design to store failed CRR as keys.

Alternatives considered:

Setting the hash fields with a timestamp and then retroactively removing hashes at certain time intervals.
Setting hash keys with a timestamp, one for each expiry time interval, then retroactively deleting old hashes.

I considered this the simplest solution because it leverages Redis' built-in expiry and we obviate the use of a cron job to delete old keys.

This PR also fixes a problem where when >= 1000 keys are stored, the listing of specific version failures does not iterate beyond the first count. The solution is to use a recursive scan listing.

ironman-machine · 2018-05-10T21:19:09Z

PR has been updated. Reviewers, please be cautious.

jonathan-gramain · 2018-05-11T00:25:52Z

extensions/replication/ReplicationConfigValidator.js

@@ -50,6 +52,7 @@ const joiSchema = {
    },
    topic: joi.string().required(),
    replicationStatusTopic: joi.string().required(),
+    monitorReplicationFailureExpiry: joi.number().default(CRR_FAILURE_EXPIRY),


I suggest adding a S at the end of the param name to tell we expect a number of seconds (which also implicitly tells it's a time interval which is not obvious otherwise, or may be renamed to replicationFailureEntryExpiryTimeS - not a fan of monitor in the name, I find it a bit confusing).

Okay, I was following the config field monitorReplicationFailure which is forthcoming. I might suggest updating to monitorReplicationFailureExpiryTimeS so it's clear these fields reference the same functionality, but I'm not especially wedded to any naming scheme.

I may have written this comment originally before we decided to use monitorReplication... naming for the conf vars, and left it as pending for some time, so I'm good keeping consistency across config.

monitorReplicationFailureExpiryTimeS it is!

jonathan-gramain · 2018-05-11T00:28:27Z

extensions/replication/replicationStatusProcessor/ReplicationStatusProcessor.js

+                    `${bucket}:${objectKey}:${versionId}:${site}`;
+                const value = JSON.stringify(JSON.parse(kafkaEntry.value));
+                const expiry = this.repConfig.monitorReplicationFailureExpiry;
+                cmds.push(['set', key, value, 'EX', expiry]);
            }
            return undefined;


That line looks unnecessary

I believe this is to make the linter happy. 😄

The inner forEach function does not seem to return anything elsewhere, so I'm suspecting this is a left-over from a previous version that required it, but I may be wrong.

This was removed during the previous refactor 🍾, so either way!

jonathan-gramain · 2018-05-11T00:32:34Z

extensions/replication/replicationStatusProcessor/ReplicationStatusProcessor.js

-                fields.push(field, JSON.stringify(value));
+                const key = `${redisKeys.failedCRR}:` +
+                    `${bucket}:${objectKey}:${versionId}:${site}`;
+                const value = JSON.stringify(JSON.parse(kafkaEntry.value));


Is it needed to copy the kafkaEntry.value field? It's no big issue, just noticed it could be useless, in doubt we can leave it.

The subsequent PR will reduce this to just storing the source role to get the failed object's metadata.

jonathan-gramain · 2018-05-11T00:50:22Z

lib/api/BackbeatAPI.js

+            if (allKeys.length >= 1000 || Number.parseInt(cursor, 10) === 0) {
+                return cb(null, cursor, allKeys);
+            }
+            return this._scanAllKeys(pattern, cursor, allKeys, cb);


I'm not familiar with redis API, but if asked for 1000 keys, may it return you less than this even if there are 1000+ keys to return? In such case it looks correct to call _scanAllKeys again, otherwise it seems we could just return the results in the callback in all cases.

There isn't a guarantee on the number of keys being returned by Redis during the SCAN operation. (I've added a point about this in the Operational Considerations section of the doc for this feature.) We do have a guarantee that the scan has completed when the returned cursor is 0, so in such a case we can return whatever results are there even if less than 1000.

jonathan-gramain · 2018-05-29T17:30:08Z

lib/api/BackbeatAPI.js

@@ -378,11 +395,12 @@ class BackbeatAPI {
                    LastModified: queueEntry.getLastModified(),
                    ReplicationStatus: 'PENDING',
                });
-                const field = `${Bucket}:${Key}:${VersionId}:${StorageClass}`;
-                return this._deleteFailedCRRField(field, err => {
+                const key = `${redisKeys.failedCRR}:` +


It would be ideal if the construction of the key happened in a shared helper function

jonathan-gramain · 2018-05-29T17:46:49Z

tests/functional/api/BackbeatServer.js

-        return ['hset', REDIS_KEY_FAILED_CRR, field, value];
+        const [bucket, objectKey, versionId, site] = key.split(':');
+        const value = getKafkaEntry(bucket, objectKey, site);
+        return ['set', `${REDIS_KEY_FAILED_CRR}:${key}`, value];


Is REDIS_KEY_FAILED_CRR the same than the redisKeys.failedCRR constant used elsewhere? In which case better use the same one consistently and remove the other.

This is used because the test suite is not run with the TEST_SWITCH environment var. I updated the constant variable name to TEST_REDIS_KEY_FAILED_CRR to make that more apparent.

Maybe we want to update that in the CI? I'm fine either way.

jonathan-gramain · 2018-05-29T17:48:14Z

tests/functional/api/BackbeatServer.js

                            const [bucket, key, versionId, site] = k.split(':');
                            assert(res.Versions.some(o => (
                                o.Bucket === bucket &&
                                o.Key === key &&
-                                o.VersionId === versionId &&
+                                o.VersionId === testVersionId &&


Why is it needed (or better) to change this test?

Because the response gets the version ID using the object MD instead of the key name (see change here. I think it's better because it actually checks that the returned value is the correct version ID. I will go ahead an update the key names to use testVersionId just to be consistent.

ironman-machine · 2018-05-29T22:16:41Z

PR has been updated. Reviewers, please be cautious.

philipyoo · 2018-05-29T22:05:02Z

lib/api/BackbeatAPI.js

-            return cb(null, this._getFailedCRRResponse(cursor, hashes));
+            const [cursor, keys] = collection;
+            allKeys.push(...keys);
+            if (allKeys.length >= 1000 || Number.parseInt(cursor, 10) === 0) {


allKeys.length >= 1000
If we get 1000 keys initially, won't this return before getting next set of keys? Or was this intentional?

Yes, in effect it allows for a paginated response. I wanted to limit the API response listing to ~1000 keys to model closer to the default for version listings in S3. We cannot guarantee the exact number because Redis does not make a guarantee for the number of keys returned during the scan. So if it's 1000 or more, the API will include a NextMarker value for subsequent listings.

Do it again human slave!:point_right: :runner: (Oh and the pull request has been updated, by the way.)

ironman-machine · 2018-05-29T22:54:01Z

PR has been updated. Reviewers, please be cautious.

ironman-machine · 2018-05-29T22:54:01Z

CONFLICT (add/add): Merge conflict in tests/unit/lifecycle/LifecycleTask.spec.js
CONFLICT (add/add): Merge conflict in tests/functional/api/BackbeatServer.js
CONFLICT (add/add): Merge conflict in lib/api/BackbeatAPI.js
CONFLICT (add/add): Merge conflict in extensions/replication/replicationStatusProcessor/ReplicationStatusProcessor.js
CONFLICT (add/add): Merge conflict in extensions/replication/ReplicationConfigValidator.js
CONFLICT (add/add): Merge conflict in extensions/lifecycle/tasks/LifecycleTask.js

ironman-machine · 2018-05-29T23:02:43Z

PR has been updated. Reviewers, please be cautious.

ironman-machine · 2018-05-29T23:03:57Z

PR has been updated. Reviewers, please be cautious.

ironman-machine · 2018-05-29T23:37:32Z

PR has been updated. Reviewers, please be cautious.

ironman-machine · 2018-05-29T23:37:32Z

CONFLICT (add/add): Merge conflict in tests/functional/api/BackbeatServer.js
CONFLICT (add/add): Merge conflict in lib/api/BackbeatAPI.js
CONFLICT (add/add): Merge conflict in extensions/replication/replicationStatusProcessor/ReplicationStatusProcessor.js
CONFLICT (add/add): Merge conflict in extensions/replication/ReplicationConfigValidator.js

ironman-machine · 2018-05-29T23:40:49Z

PR has been updated. Reviewers, please be cautious.

ironman-machine · 2018-05-29T23:45:40Z

PR has been updated. Reviewers, please be cautious.

ironman-machine · 2018-05-30T00:02:01Z

PR has been updated. Reviewers, please be cautious.

ironman-machine · 2018-05-31T18:37:53Z

PR has been updated. Reviewers, please be cautious.

ironman-machine · 2018-05-31T18:37:53Z

CONFLICT (add/add): Merge conflict in package.json

bennettbuchanan force-pushed the ft/ZENKO-147/useRedisKeys branch from 9f4f82a to d4fa76f Compare May 10, 2018 21:19

jonathan-gramain reviewed May 29, 2018

View reviewed changes

bennettbuchanan force-pushed the ft/ZENKO-147/useRedisKeys branch from d4fa76f to 30ed3fb Compare May 29, 2018 22:16

philipyoo previously approved these changes May 29, 2018

View reviewed changes

bennettbuchanan force-pushed the ft/ZENKO-147/useRedisKeys branch from a92bc62 to c8513a6 Compare May 29, 2018 23:02

bennettbuchanan force-pushed the ft/ZENKO-147/useRedisKeys branch from c8513a6 to d67f3a0 Compare May 29, 2018 23:03

bennettbuchanan force-pushed the ft/ZENKO-147/useRedisKeys branch from b1e203c to ee0e938 Compare May 29, 2018 23:40

bennettbuchanan force-pushed the ft/ZENKO-147/useRedisKeys branch from ee0e938 to 63aa34d Compare May 29, 2018 23:45

bennettbuchanan force-pushed the ft/ZENKO-147/useRedisKeys branch from 63aa34d to 1341403 Compare May 30, 2018 00:01

ft: ZENKO-147 Use Redis keys instead of hash

2da08f9

bennettbuchanan force-pushed the ft/ZENKO-147/useRedisKeys branch from 1341403 to 2da08f9 Compare May 31, 2018 18:37

bennettbuchanan changed the base branch from z/1.0 to development/8.0 May 31, 2018 18:38

philipyoo approved these changes May 31, 2018

View reviewed changes

jonathan-gramain approved these changes May 31, 2018

View reviewed changes

jonathan-gramain merged commit f3f7a0d into development/8.0 May 31, 2018

jonathan-gramain deleted the ft/ZENKO-147/useRedisKeys branch May 31, 2018 18:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ft: ZENKO-147 Use Redis keys instead of hash #286

ft: ZENKO-147 Use Redis keys instead of hash #286

bennettbuchanan commented May 10, 2018 •

edited

Loading

ironman-machine commented May 10, 2018

jonathan-gramain May 11, 2018

bennettbuchanan May 29, 2018

jonathan-gramain May 29, 2018

bennettbuchanan May 29, 2018

jonathan-gramain May 11, 2018

bennettbuchanan May 29, 2018

jonathan-gramain May 29, 2018

bennettbuchanan May 29, 2018

jonathan-gramain May 11, 2018

bennettbuchanan May 29, 2018

jonathan-gramain May 11, 2018

bennettbuchanan May 29, 2018

jonathan-gramain May 29, 2018

jonathan-gramain May 29, 2018

bennettbuchanan May 30, 2018

bennettbuchanan May 30, 2018

jonathan-gramain May 29, 2018

bennettbuchanan May 29, 2018

ironman-machine commented May 29, 2018

philipyoo May 29, 2018

bennettbuchanan May 29, 2018 •

edited

Loading

ironman-machine commented May 29, 2018

ironman-machine commented May 29, 2018

ironman-machine commented May 29, 2018

ironman-machine commented May 29, 2018

ironman-machine commented May 29, 2018

ironman-machine commented May 29, 2018

ironman-machine commented May 29, 2018

ironman-machine commented May 29, 2018

ironman-machine commented May 30, 2018

ironman-machine commented May 31, 2018

ironman-machine commented May 31, 2018

ft: ZENKO-147 Use Redis keys instead of hash #286

ft: ZENKO-147 Use Redis keys instead of hash #286

Conversation

bennettbuchanan commented May 10, 2018 • edited Loading

ironman-machine commented May 10, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ironman-machine commented May 29, 2018

Choose a reason for hiding this comment

bennettbuchanan May 29, 2018 • edited Loading

Choose a reason for hiding this comment

ironman-machine commented May 29, 2018

ironman-machine commented May 29, 2018

ironman-machine commented May 29, 2018

ironman-machine commented May 29, 2018

ironman-machine commented May 29, 2018

ironman-machine commented May 29, 2018

ironman-machine commented May 29, 2018

ironman-machine commented May 29, 2018

ironman-machine commented May 30, 2018

ironman-machine commented May 31, 2018

ironman-machine commented May 31, 2018

bennettbuchanan commented May 10, 2018 •

edited

Loading

bennettbuchanan May 29, 2018 •

edited

Loading