Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added Pulsar IO connector for local files #2869

Merged
merged 8 commits into from
Dec 27, 2018

Conversation

david-streamlio
Copy link
Contributor

Motivation

Added a Pulsar IO connector for consuming files from the local filesystem

Modifications

Added a new module to the pulsar-io module that includes the Pulsar file connector and its associated classes & tests

Result

After your change, users will be able to consume files from the local filesystem, and have the contents directly published to a Pulsar topic.

@sijie sijie assigned sijie and david-streamlio and unassigned sijie Oct 28, 2018
@sijie sijie added type/feature The PR added a new feature or issue requested a new feature area/connector labels Oct 28, 2018
@jiazhai
Copy link
Member

jiazhai commented Nov 6, 2018

rerun java8 tests

@jiazhai
Copy link
Member

jiazhai commented Nov 6, 2018

overall lgtm, But @david-streamlio Seems the ut is flaky: org.apache.pulsar.io.file.FileSourceConfigTests.InvalidBooleanPropertyTest
https://builds.apache.org/job/pulsar_precommit_java8/4711/

@david-streamlio
Copy link
Contributor Author

@sijie I am not sure why the test fails during testing. I am able to run the test successfully locally using TestNG 6.13.1. Do we use a different version of TestNG on Jenkins?

The error is related to the TestNG regex not matching the expected string, but as you can see from visual inspection that the String should match.

"@test(expectedExceptions = com.fasterxml.jackson.databind.exc.InvalidFormatException.class,
expectedExceptionsMessageRegExp = "Can not deserialize value of type java.lang.Boolean from String .*")

@david-streamlio
Copy link
Contributor Author

rerun java8 tests

@sijie
Copy link
Member

sijie commented Nov 22, 2018

@david-streamlio can you rebase this PR to latest master?

@sijie sijie added this to the 2.3.0 milestone Dec 27, 2018
@sijie sijie merged commit 209c448 into apache:master Dec 27, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/connector type/feature The PR added a new feature or issue requested a new feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants