Skip to content

Tests for the to_kafka stream. #236

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 18 commits into from
Closed

Tests for the to_kafka stream. #236

wants to merge 18 commits into from

Conversation

@chinmaychandak
Copy link
Contributor

@chinmaychandak chinmaychandak commented Mar 25, 2019

Added to_kafka_batched() as a DaskStream API, and added tests for to_kafka_batched() w/ and w/o Dask.

jsmaupin and others added 18 commits January 8, 2019 18:22
@jsmaupin
added to_kafka stream
80efa35
@jsmaupin
remove call to _global_sinks.add(self) as it is wrong and not needed
b635167
@jsmaupin
Locally import Confluent Kafka library
893300f
@jsmaupin
need to wait for subscription to happen before producing messages. see:
3b546f3 
confluentinc/confluent-kafka-python#156 (comment)
@jsmaupin
from_kafka_batched does not return the Kafka stream directly. The sto…
2802078 
…pped parameter is in the upstream
@jsmaupin
Removed atexit use. Replaced it with a provided .flush() method. Adde…
ae7a6f2 
…d a test.
@jsmaupin
move blocking subscribe to another function
3a808ae
@jsmaupin
fixed documentation
a1f0b09
@jsmaupin
remove unusued parameters
a1104b6
@jsmaupin
add sleep to wait for the subscribe() operation to complete
c6aea02
@jsmaupin
add future to to_kafka update() to allow client ot feel backpressure
fd7ed40
@jsmaupin
Rely on Confluent lib's limited queue size for back-pressure
7e0efc1
@jsmaupin
added to_kafka_batched(). Docstrings and tests still incoming.
bf68b79
@jsmaupin
apply patch to fix flake8 errors
d206a69
@jsmaupin
Merge branch 'master' into master
04dbf77
@jsmaupin
some E811 noqa's shoudl be F811
1ae904d
@jsmaupin
Fix test to be Py2 compatible. Use tornado gen/yield pattern.
62d523b
Added to_kafka_batched() onto DaskStream APIs, and added pertinent te…
7c9382b 
…sts.
@martindurant
Copy link
Member

martindurant commented Mar 25, 2019

Most of this was in #230
The batched part was specifically removed there, as it provided no backpressure.
If you intend to push this, please rebase on master (all those commits and conflicts should not show up) and start again, making clear what is new.

@chinmaychandak chinmaychandak deleted the to_kafka_batched_dask branch April 3, 2020 17:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@chinmaychandak @martindurant @jsmaupin