ACCUMULO-4501 (work in progress) Very simple implementation of a visib… #168
Conversation
|
One option to consider instead of modifying RFile is to make it a decorator like BloomFilterLayer. BloomFilterLayer stores its information in RFile metadata. I'm think will be problems with this approach, but I would not know what they are w/o actually trying it. Are you considering making this a generic histogram functionality, where the user can configure a function that emits counts for a given Key Value? |
| public long increment(Key k) { | ||
| final Text t = buffer.get(); | ||
| Objects.requireNonNull(k.getColumnVisibility(t)); | ||
| AtomicLong count = histogram.get(t); |
There was a problem hiding this comment.
Is an AtomicLong needed for concurrency reasons? If not, then could create a simple class like MutableLong in MapCounter to avoid volatile.
There was a problem hiding this comment.
I don't think it would be needed. Using something faster would probably be better.
Sam applies to using an unbounded HashMap for storage (e.g. handling tables with 100's to 1000's of visibilities per file).
| public void append(Key key, Value value) throws IOException { | ||
| Text _text = buffer.get(); | ||
| key.getColumnVisibility(_text); | ||
| AtomicLong count = histogram.get(_text); |
There was a problem hiding this comment.
I think this could use the ByteSequence returned by getColumnVisibilityData() to do the lookup in the map (would require keying map on ByteSequence). This would avoid the copy for lookup. Only copy when inserting into the map.
|
|
||
| public void append(Key key, Value value) throws IOException { | ||
| Text _text = buffer.get(); | ||
| key.getColumnFamily(_text); |
There was a problem hiding this comment.
get the family? Is this duplicate code?
There was a problem hiding this comment.
Lol, bad copy-paste :)
This is my first foray into the RFile codebase, so I am very happy to be redirected into a different implementation :). My liberal use of increasing visibility on classes ought to be apparent haha.
If we can abstract this specific feature into something more generic without it blowing up, I'm ok with that. I just don't have a big picture view right now. |
|
@joshelser I have not forgotten about this. We had discussed writing up a design doc on IRC. I have just been busy. I plan to take a crack at that Thur. |
|
@keith-turner nbd. This has fallen by the way-side for me too :) |
|
I just posted #180 for review. |
2 participants
…ility histogram