kvs: add flux_kvs_lookup_get_raw() #1218

garlick · 2017-09-28T22:15:24Z

This PR adds flux_kvs_lookup_get_raw() to the KVS API, to go with flux_kvs_txn_put_raw(), then modifies t/kvs/basic to use them instead of an extra base64 encoding layer for copy-tokvs and copy-fromkvs subcommands. Finally add a couple of sharness tests to fill test gaps identified in #1214 (lookup unencoded content) and #1215 (zero length value).

This is based on top of #1214

chu11 · 2017-09-28T22:51:26Z

LGTM, will hit button after it passes.

chu11 · 2017-09-28T22:55:05Z

actually, one nit, I have a # TODO - convert to using "flux content store", see issue1216 comment in t1002-kvs-extra.t. How about axing that test and replacing it with the one you just wrote.

garlick · 2017-09-28T23:09:13Z

Ah sorry I missed that. That commit is now just a one line change to the test (and removal of the TODO).

garlick · 2017-09-28T23:11:09Z

Hang on typo in that one, fix coming.

coveralls · 2017-09-28T23:13:07Z

Coverage increased (+0.01%) to 78.639% when pulling 0be8dff on garlick:raw_api into 19d06f1 on flux-framework:master.

grondo · 2017-09-28T23:19:05Z

Does the new API function need an entry in flux_kvs_lookup.adoc?

garlick · 2017-09-28T23:21:34Z

Yes :-/

coveralls · 2017-09-28T23:34:02Z

Coverage increased (+0.008%) to 78.635% when pulling 464af50 on garlick:raw_api into 19d06f1 on flux-framework:master.

codecov-io · 2017-09-28T23:34:51Z

Codecov Report

Merging #1218 into master will decrease coverage by <.01%.
The diff coverage is 86.66%.

@@            Coverage Diff             @@
##           master    #1218      +/-   ##
==========================================
- Coverage   78.21%   78.21%   -0.01%     
==========================================
  Files         158      158              
  Lines       29283    29297      +14     
==========================================
+ Hits        22904    22914      +10     
- Misses       6379     6383       +4

Impacted Files	Coverage Δ
src/common/libkvs/kvs_txn.c	`78.14% <ø> (ø)`	⬆️
src/common/libkvs/kvs_lookup.c	`74.1% <86.66%> (+1.94%)`	⬆️
src/common/libutil/blobref.c	`97.22% <0%> (-1.39%)`	⬇️
src/common/libflux/request.c	`87.17% <0%> (-1.29%)`	⬇️
src/common/libflux/rpc.c	`93.38% <0%> (-0.83%)`	⬇️
src/broker/content-cache.c	`73.43% <0%> (-0.65%)`	⬇️
src/common/libflux/message.c	`81.25% <0%> (-0.36%)`	⬇️
src/common/libcompat/rpc.c	`94.57% <0%> (-0.05%)`	⬇️
src/common/libflux/future.c	`88.31% <0%> (+0.46%)`	⬆️
src/common/libutil/base64.c	`95.77% <0%> (+0.7%)`	⬆️
... and 4 more

coveralls · 2017-09-28T23:54:59Z

Coverage increased (+0.0007%) to 78.628% when pulling 464af50 on garlick:raw_api into 19d06f1 on flux-framework:master.

garlick · 2017-09-29T00:05:47Z

OK, man pages all straightened out for these functions, plus fixed some Makefile.am errors in the man3 directory (broken out to separate commits)

coveralls · 2017-09-29T00:22:22Z

Coverage increased (+0.01%) to 78.639% when pulling 86d091a on garlick:raw_api into 19d06f1 on flux-framework:master.

chu11 · 2017-09-29T02:46:52Z

doc/man3/flux_kvs_txn_create.adoc

@@ -31,6 +31,10 @@ SYNOPSIS
 int flux_kvs_txn_symlink (flux_kvs_txn_t *txn, int flags,
                           const char *key, const char *target);

+ int flux_kvs_txn_put_raw (flux_kvs_txn_t *txn, int flags,
+                           const char *key, void *data, int len);
+


const void *data?

I think you might be looking at an older version there? This is already done.

I probably shouldn't have forced pushed a squashed version back there.

still shows up void * to me. This is the adoc, not the header file.

chu11 · 2017-09-29T02:49:18Z

also flux_kvs_lookup_get_raw() can be added to libkvs/test/kvs_lookup.c to capture the bad input case.

coveralls · 2017-09-29T14:46:09Z

Coverage increased (+0.07%) to 78.699% when pulling f989abb on garlick:raw_api into 19d06f1 on flux-framework:master.

garlick · 2017-09-29T14:53:49Z

Ah thanks, just pushed a fix.

coveralls · 2017-09-29T15:19:27Z

Coverage decreased (-0.006%) to 78.621% when pulling 6b7e8b6 on garlick:raw_api into 19d06f1 on flux-framework:master.

garlick · 2017-09-29T15:29:23Z

Restarted one builder due to #1145 write error.

garlick · 2017-09-29T16:57:02Z

@chu11 and @grondo, if you're OK with this I'll squash it.

chu11 · 2017-09-29T17:30:22Z

looks good to me, squash away

Call flux_kvs_lookup_get_raw() with bad parameters and make sure it fails with errno == EINVAL.

Problem: copy-tokvs and copy-fromkvs use custom JSON encapsulated base64 encoding to store raw data. Use flux_kvs_txn_put_raw() and flux_kvs_lookup_get_raw().

Add test that creates a 'valref' entry in the KVS pointing to content stored out of band. Read the raw content back in with flux_kvs_lookup_get_raw(). Fixes flux-framework#1216

Store a zero length raw value and read it back. Fixes flux-framework#1215

Problem: Makefile.am dependencies were not updated when lookup functions were renamed. Fix names.

Problem: Makefile.am dependencies were not added for flux_kvs_txn_symlink(3). Add the dependency.

garlick · 2017-09-29T18:04:19Z

Thanks, squashed.

chu11 · 2017-09-29T18:23:41Z

LGTM, will hit button after passes travis

coveralls · 2017-09-29T18:24:16Z

Coverage decreased (-0.003%) to 78.624% when pulling 5326555 on garlick:raw_api into 19d06f1 on flux-framework:master.

garlick force-pushed the raw_api branch from 4901d4d to 0be8dff Compare September 28, 2017 22:16

libkvs/lookup: add flux_kvs_lookup_get_raw()

0184f3b

garlick force-pushed the raw_api branch from 0be8dff to 2d85019 Compare September 28, 2017 23:08

garlick force-pushed the raw_api branch from 2d85019 to 464af50 Compare September 28, 2017 23:15

garlick force-pushed the raw_api branch from 464af50 to 86d091a Compare September 29, 2017 00:00

libkvs/txn: make flux_kvs_txn_put_raw() arg const

7b4f894

chu11 reviewed Sep 29, 2017

View reviewed changes

garlick added 4 commits September 29, 2017 18:03

libkvs/test: cover flux_kvs_lookup_get_raw() EINVAL

37933a0

Call flux_kvs_lookup_get_raw() with bad parameters and make sure it fails with errno == EINVAL.

t/kvs/basic: copy-* subcommand use raw API

3e6a35e

Problem: copy-tokvs and copy-fromkvs use custom JSON encapsulated base64 encoding to store raw data. Use flux_kvs_txn_put_raw() and flux_kvs_lookup_get_raw().

t/kvs: add test for raw content

42f1ee6

Add test that creates a 'valref' entry in the KVS pointing to content stored out of band. Read the raw content back in with flux_kvs_lookup_get_raw(). Fixes flux-framework#1216

t/kvs: test zero size raw values

275537e

Store a zero length raw value and read it back. Fixes flux-framework#1215

garlick added 4 commits September 29, 2017 18:03

doc/flux_kvs_lookup(3): fix man aliases

183091b

Problem: Makefile.am dependencies were not updated when lookup functions were renamed. Fix names.

doc/flux_kvs_txn_create(3): fix man aliases

a63c79e

Problem: Makefile.am dependencies were not added for flux_kvs_txn_symlink(3). Add the dependency.

doc/flux_kvs_txn_create(3): add flux_kvs_txn_put_raw()

d446189

doc/flux_kvs_lookup(3): add flux_kvs_lookup_get_raw()

5326555

garlick force-pushed the raw_api branch from 6b7e8b6 to 5326555 Compare September 29, 2017 18:03

chu11 merged commit 96a165c into flux-framework:master Sep 29, 2017

garlick deleted the raw_api branch October 2, 2017 16:42

grondo mentioned this pull request May 10, 2018

0.9.0 Release #1479

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kvs: add flux_kvs_lookup_get_raw() #1218

kvs: add flux_kvs_lookup_get_raw() #1218

garlick commented Sep 28, 2017

chu11 commented Sep 28, 2017

chu11 commented Sep 28, 2017

garlick commented Sep 28, 2017

garlick commented Sep 28, 2017

coveralls commented Sep 28, 2017

grondo commented Sep 28, 2017

garlick commented Sep 28, 2017

coveralls commented Sep 28, 2017

codecov-io commented Sep 28, 2017 •

edited

coveralls commented Sep 28, 2017

garlick commented Sep 29, 2017

coveralls commented Sep 29, 2017

chu11 Sep 29, 2017

garlick Sep 29, 2017

chu11 Sep 29, 2017

chu11 commented Sep 29, 2017 •

edited

coveralls commented Sep 29, 2017

garlick commented Sep 29, 2017

coveralls commented Sep 29, 2017

garlick commented Sep 29, 2017

garlick commented Sep 29, 2017

chu11 commented Sep 29, 2017

garlick commented Sep 29, 2017

chu11 commented Sep 29, 2017

coveralls commented Sep 29, 2017

kvs: add flux_kvs_lookup_get_raw() #1218

kvs: add flux_kvs_lookup_get_raw() #1218

Conversation

garlick commented Sep 28, 2017

chu11 commented Sep 28, 2017

chu11 commented Sep 28, 2017

garlick commented Sep 28, 2017

garlick commented Sep 28, 2017

coveralls commented Sep 28, 2017

grondo commented Sep 28, 2017

garlick commented Sep 28, 2017

coveralls commented Sep 28, 2017

codecov-io commented Sep 28, 2017 • edited

Codecov Report

coveralls commented Sep 28, 2017

garlick commented Sep 29, 2017

coveralls commented Sep 29, 2017

chu11 Sep 29, 2017

Choose a reason for hiding this comment

garlick Sep 29, 2017

Choose a reason for hiding this comment

chu11 Sep 29, 2017

Choose a reason for hiding this comment

chu11 commented Sep 29, 2017 • edited

coveralls commented Sep 29, 2017

garlick commented Sep 29, 2017

coveralls commented Sep 29, 2017

garlick commented Sep 29, 2017

garlick commented Sep 29, 2017

chu11 commented Sep 29, 2017

garlick commented Sep 29, 2017

chu11 commented Sep 29, 2017

coveralls commented Sep 29, 2017

codecov-io commented Sep 28, 2017 •

edited

chu11 commented Sep 29, 2017 •

edited