Make Uploader pluggable through secor.properties#207
Make Uploader pluggable through secor.properties#207HenryCaiHaiying merged 1 commit intopinterest:masterfrom
Conversation
lucamilanesio
force-pushed
the
pluggable-uploader-policy
branch
2 times, most recently
from
dac99a8 to
5af43c8
Compare
| package com.pinterest.secor.common; | ||
|
|
||
| import com.google.api.client.repackaged.com.google.common.base.Strings; | ||
| import com.pinterest.secor.uploader.Uploader; |
There was a problem hiding this comment.
Doesn't quite like this added dependency to secor.uploader package, I think you can have this property have a default value in secor.common.properties (there are quite a few config property specifying classname in secor.common.properties already)
There was a problem hiding this comment.
@HenryCaiHaiying thanks for the feedback, makes sense, will include in my next amended commit.
lucamilanesio
force-pushed
the
pluggable-uploader-policy
branch
from
5af43c8 to
16f95b9
Compare
|
@HenryCaiHaiying all changes you requested have been applied, could you have a second look please? Thank you in advance. Luca. |
| UploadManager uploadManager) { | ||
| this(config, offsetTracker, fileRegistry, uploadManager, | ||
| new ZookeeperConnector(config)); | ||
|
|
There was a problem hiding this comment.
Can you add the javadoc to this 'init' method?
There was a problem hiding this comment.
Will do, even though there was no JavaDoc on the previous constructor ... but I can deduct what is what from the code :-)
lucamilanesio
force-pushed
the
pluggable-uploader-policy
branch
from
16f95b9 to
7c1daa1
Compare
Allow custom logic to be applied to flush locally stored topic files to the underlying storage system.
lucamilanesio
force-pushed
the
pluggable-uploader-policy
branch
from
7c1daa1 to
ca0cf23
Compare
|
@HenryCaiHaiying thanks for the extra reviews, amended the commit and rebased on current master. |
|
Build is green, hope we should be good to go 😄 |
|
Looks good, thanks for the contribution. |
2 participants
Allow custom logic to be applied to flush locally stored topic files
to the underlying storage system.
Currently, only a single logic can be applied, and it is coded in the Uploader class.
By making the class pluggable and configurable through secor.upload.class property
we can just override the applyPolicy() and customize the logic on how to flush data.
This change is fully backward compatible. When property is not configured the Uploader
class is instantiated as before.