-
Notifications
You must be signed in to change notification settings - Fork 3.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Added Pulsar IO connector for local files #2869
Conversation
pulsar-io/file/src/main/java/org/apache/pulsar/io/file/FileConsumerThread.java
Outdated
Show resolved
Hide resolved
pulsar-io/file/src/main/java/org/apache/pulsar/io/file/FileSource.java
Outdated
Show resolved
Hide resolved
pulsar-io/file/src/test/java/org/apache/pulsar/io/file/AbstractFileTests.java
Show resolved
Hide resolved
rerun java8 tests |
overall lgtm, But @david-streamlio Seems the ut is flaky: org.apache.pulsar.io.file.FileSourceConfigTests.InvalidBooleanPropertyTest |
@sijie I am not sure why the test fails during testing. I am able to run the test successfully locally using TestNG 6.13.1. Do we use a different version of TestNG on Jenkins? The error is related to the TestNG regex not matching the expected string, but as you can see from visual inspection that the String should match. "@test(expectedExceptions = com.fasterxml.jackson.databind.exc.InvalidFormatException.class, |
rerun java8 tests |
@david-streamlio can you rebase this PR to latest master? |
Motivation
Added a Pulsar IO connector for consuming files from the local filesystem
Modifications
Added a new module to the pulsar-io module that includes the Pulsar file connector and its associated classes & tests
Result
After your change, users will be able to consume files from the local filesystem, and have the contents directly published to a Pulsar topic.