Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release version 1.1.0 #71

Merged
merged 19 commits into from
Jul 6, 2020
Merged

Release version 1.1.0 #71

merged 19 commits into from
Jul 6, 2020

Conversation

mmolimar
Copy link
Owner

@mmolimar mmolimar commented Jul 6, 2020

No description provided.

mmolimar and others added 19 commits May 20, 2020 08:37
Fix the text file reader section.
* Add a simple batch policy for file systems with a lot of files

* Improve code test coverage

* Setup batching in the AbstractPolicy instead of a seperate Policy.

* Fix unit tests

* Fix unit tests by isolating newly created policies

* Rename iterator

* Use com.google.common.collect.Iterators.partition method instead

* Minor changes

Co-authored-by: Mario Molina <mmolimar@gmail.com>
* Skip processing unchanged files

We store the fileSizeBytes as an offset in kafka connect and skip files whose file size has not changed

* Add tests

* Reduce log line length

* Add EmptyFileReader

* Update tests

* Minor changes

Co-authored-by: Mario Molina <mmolimar@gmail.com>
* Make makeReader run only if reading the file is necessary

This is really important because if we open up a filehandle for reading some hadoop filesystem implementations (like s3a) will open up a connection per file and these connections will get exhausted quickly

* Handle 'null' from the offsets map

* handle nulls better

* Minor changes

Co-authored-by: Mario Molina <mmolimar@gmail.com>
@coveralls
Copy link

Coverage Status

Coverage increased (+0.7%) to 96.249% when pulling 25890da on develop into 588d310 on master.

@mmolimar mmolimar merged commit 7ec9293 into master Jul 6, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants