Add CI/CD for StreamingDataFrames #189
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
daniil-quix
force-pushed
the
chore/sdf-ci-cd
branch
from
026003a
to
af7edbc
Compare
- Re-write Consumer tests without threading - In Runner tests, execute Runner code in the same thread - Add debug logs to tests
Relax the "requests" version
- Keep pytest and black settings in pyproject.toml - Use pre-commit to run black before committing
Specify "packages" option
Run linters and tests on each commit if there're any changes to "src/StreamingDataFrames/**"
daniil-quix
force-pushed
the
chore/sdf-ci-cd
branch
from
0215de7
to
471611a
Compare
|
Additions and changes make sense 👍 |
tim-quix
approved these changes
Sep 28, 2023
daniil-quix
added a commit
that referenced
this pull request
Nov 7, 2023
* Update .gitignore * Add debug logs to Consumer and Producer * Add "Runner.stop" to break the internal polling loop * Update tests to run on Github Runners * Update requirements * Add "black" formatter, pyproject.toml and pre-commit * Update setup.cfg * Add CI/CD workflow for Streaming Dataframes
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Main changes
blackbefore commitNotes
The Runner and Consumer tests needed to be updated.
They were getting stuck in the CI pipeline because
Consumersometimes couldn't close itself, even though it works fine locally.Apparently,
confluent_kafka.Consumerdoesn't like being passed around between threads, and we use threads in tests.When you open Consumer in one thread, do
poll()in another thread, and then close it in the first thread, it might get stuck indefinitely.The fix is to always keep the work with the Consumer instance in the same thread where it's created.