-
Notifications
You must be signed in to change notification settings - Fork 127
Feature 256/deterministic iterator #258
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature 256/deterministic iterator #258
Conversation
Signed-off-by: kuba-- <kuba@sourced.tech>
Signed-off-by: kuba-- <kuba@sourced.tech>
Signed-off-by: kuba-- <kuba@sourced.tech>
Signed-off-by: kuba-- <kuba@sourced.tech>
Signed-off-by: kuba-- <kuba@sourced.tech>
Signed-off-by: kuba-- <kuba@sourced.tech>
LGTM after @jfontan have a look on it. |
@jfontan - I think, I can also remove internal done channel. Actually we don't use it (only internally), so I believe
If I remove
WDYT? |
@kuba-- As you say we don't need |
Signed-off-by: kuba-- <kuba@sourced.tech>
This is pretty radical change, so please review carefully. It's related to #256
What has been changed to avoid randomised order by Repository iterator:
Current implementation relied on multiple channels which were populated in constructor
NewRowRepoIter
.First we iterated over repos and push them to repos channel, at the same time N go routines pulled repos from a channel, convert them to rows and pushed rows to the next channel.
At the end we had a channel with a mix of rows from different repos (multiplexed by go routines).
This PR drastically removes lot of complexity and make everything linear in the
Next
function.We just iterate over rows till we get an
EOF
error, then we move to the next repo.I left only one
done
channel for internal usage (mainly for tests' graceful shutdown) andcontext
.Also what's worth to notice are
atomic
operations inRepositoryIter.Next
function to make the API thread-safe