Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: build wmstore and wmstorewatcher directly, and remove some unnecessary fields #970

Merged
merged 5 commits into from
Aug 21, 2023

Conversation

whynowy
Copy link
Member

@whynowy whynowy commented Aug 20, 2023

  1. Functions to build WatermarkStore and WatermarkStoreWatcher directly from bucket.
  2. Remove bucket (only for logging) as an arg, use ctx to pass it.
  3. Remove pipeline (only for logging) as an arg, use ctx to pass it.

Signed-off-by: Derek Wang <whynowy@gmail.com>
Signed-off-by: Derek Wang <whynowy@gmail.com>
Signed-off-by: Derek Wang <whynowy@gmail.com>
Signed-off-by: Derek Wang <whynowy@gmail.com>
@whynowy whynowy requested a review from vigith as a code owner August 20, 2023 06:25
bucketName: bucketName,
kvEntryCh: kvEntryCh,
kvHistory: []kvs.KVEntry{},
updatesChMap: make(map[string]chan kvs.KVEntry),
doneCh: make(chan struct{}),
log: logging.FromContext(ctx).With("pipeline", pipelineName).With("bucketName", bucketName),
log: logging.FromContext(ctx).With("bucketName", bucketName),
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only for logging, removed.

var err error
var jsStore = &jetStreamStore{
client: client,
log: logging.FromContext(ctx).With("pipeline", pipelineName).With("kvName", kvName),
log: logging.FromContext(ctx).With("kvName", kvName),
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The log in ctx comes with pipeline.

@@ -66,7 +66,7 @@ func NewKVJetStreamKVWatch(ctx context.Context, pipelineName string, kvName stri
kvwTimer: time.NewTimer(kvOpts.watcherCreationThreshold),
opts: kvOpts,
doneCh: make(chan struct{}),
log: logging.FromContext(ctx).With("pipeline", pipelineName).With("kvName", kvName),
log: logging.FromContext(ctx).With("kvName", kvName),
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The log in ctx comes with pipeline.

fromBufferPartitionCount: fromBufferPartitionCount,
log: logging.FromContext(ctx).With("bucket", bucket),
log: logging.FromContext(ctx),
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The log from the ctx comes with bucket.

// Initialize a new empty watermarks DLL with nil values of the size capacity.
// This is to avoid length check: when a new element is added, the tail element will be deleted.
offsetTimeline := OffsetTimeline{
ctx: ctx,
capacity: c,
log: logging.FromContext(ctx).With("bucket", bucket),
log: logging.FromContext(ctx),
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The log already has bucket.

@whynowy whynowy changed the title refactor: hide hb/ot stores and remove some unnecessary fields refactor: build wmstore and wmstorewatcher directly, and remove some unnecessary fields Aug 20, 2023
.
Signed-off-by: Derek Wang <whynowy@gmail.com>
@yhl25 yhl25 merged commit 1f33bf8 into numaproj:main Aug 21, 2023
16 checks passed
@whynowy whynowy deleted the reem branch August 21, 2023 04:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants