Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

libutil: add Flux Local Unique ID (FLUID) implementation #1541

Merged
merged 3 commits into from May 29, 2018

Conversation

Projects
None yet
3 participants
@garlick
Copy link
Member

garlick commented May 29, 2018

This re-proposes @grondo's work on Flux Local Unique ID (FLUID), originally proposed in #470. This seems like a useful mechanism for scalably generating Flux jobid's in the new execution system.

FLUID's are lighter weight to generate than UUID's, as they do not require a random number generator, and they fit in a uint64_t as opposed to 128 bit UUID's. Generation is distributed to up to 2^14 "generators". FLUID's are partially time-ordered. See #470 for the full discussion.

Also included is a 16K word mnemonic generator that can convert a FLUID to a 6-word phrase and back again, potentially useful for hotline communication. A fluid_encode() / fluid_decode() interface performs this and other types of string encoding.

Another encoding type was also introduced that is 4x4-digit hex numbers seprated by periods, e.g. xxxx.xxxx.xxxx.xxxx which might be useful for converting a FLUID to a KVS key and back, spreading the key space over multiple KVS directories like we do now. The encoding is selectable, so we could add different encodings that map onto KVS key space if that one doesn't work out.

@grondo

This comment has been minimized.

Copy link
Contributor

grondo commented May 29, 2018

Cool!

Two quick comments:

  • typos in 0b2b7f2: jobid's and ID's should just be jobids and IDs I think.
  • Comment on mnemonic encoder mentions a 16K word dictionary, but the wordlist included is ~1630 (was 1.6K wordlist meant here?)

Really nice job turning the ids into a standalone utility library.

@garlick garlick force-pushed the garlick:fluid branch from 70430bb to 51520c1 May 29, 2018

libutil/mnemonic: add Orin Tirosch's mnemonic encoder
Mnemonic encoder allows 32 bits of data to be converted into
three words from a ~1630 word vocabulary, with words chosen
to be easy to understand over the phone and recognizable
internationally.

From original project here (now defunct):

http://web.archive.org/web/20101031205747/http://www.tothink.com/mnemonic/

This will allow 64 bit FLUIDs to be represented as a 6 word
phrase, which may them easier to communicate (especially thinking
of incident reports for problems on classified computer networks).

Originally proposed by @grondo and discussed in #470.
@garlick

This comment has been minimized.

Copy link
Member Author

garlick commented May 29, 2018

Thanks! I tweaked those commit messages and forced a push.

garlick added some commits May 24, 2018

libutil/fluid: add Flux Local Unique IDs
Add Flux Local Unique ID implementation based on
the proposal in #470.  FLUIDs are an efficient
alternative to UUIDs for per-Flux-instance 64-bit
unique IDs.  The primary use case is Flux jobids.

Started with @grondo's work in
d81baef
libutil/test: add fluid unit test
Add unit test for basic FLUID functionality.
@codecov-io

This comment has been minimized.

Copy link

codecov-io commented May 29, 2018

Codecov Report

❗️ No coverage uploaded for pull request base (master@531a47a). Click here to learn what that means.
The diff coverage is 84.35%.

@@            Coverage Diff            @@
##             master    #1541   +/-   ##
=========================================
  Coverage          ?   78.83%           
=========================================
  Files             ?      166           
  Lines             ?    30557           
  Branches          ?        0           
=========================================
  Hits              ?    24091           
  Misses            ?     6466           
  Partials          ?        0
Impacted Files Coverage Δ
src/common/libutil/fluid.c 100% <100%> (ø)
src/common/libutil/mnemonic.c 77.04% <77.04%> (ø)

@grondo grondo merged commit b1d1b71 into flux-framework:master May 29, 2018

4 checks passed

codecov/patch No report found to compare against
Details
codecov/project No report found to compare against
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
coverage/coveralls First build on master at 79.127%
Details
@grondo

This comment has been minimized.

Copy link
Contributor

grondo commented May 29, 2018

This seems reasonable to merge as is, so I went ahead and merged it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.