Skip to content

Commit

Permalink
Merge pull request confluentinc#2 from Tumblr/usielriedl/update-docs
Browse files Browse the repository at this point in the history
Expands Tumblr specific docs
  • Loading branch information
Usiel authored and GitHub Enterprise committed Dec 13, 2023
2 parents 55629d1 + 4a8b5fe commit 2b26ae1
Showing 1 changed file with 26 additions and 5 deletions.
31 changes: 26 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,37 @@

This repo is a fork of https://github.com/confluentinc/kafka-connect-hdfs.

Our custom flavor adds a new optional configuration field `flush.file.size` to allow for file rotation based on file
size (as opposed to number of records with `flush.size`).
Our custom flavor adds
- a new optional configuration field `flush.file.size` to allow for file rotation based on file
size (as opposed to number of records with `flush.size`),
- and a custom partitioner that allows mapping of Kafka topic names to different HDFS destinations
(e.g. `ls_kraken_events` to `little_sister_kraken_events`).

Additionally, we have a diff that introduces build changes (dependency fixes, Tumblr org as name), since the current
Additionally, we have a diff that introduces build changes (dependency fixes, custom version, ...), since the current
`origin/master` does not build.

### Versioning

Our version is of the following format: `<origin-version>-tumblr+<tumblr-version>` - thus `<origin-version>` tells us
at which (version) tag we branched off and `<tumblr-version>` indicates the state of our custom flavor. Hence, if you
are adding a new customization you want to bump the `<tumblr-version>`.

### Deployment

Run `mvn package [-dskipTests]` and upload the generated zip file to [Tumblr Nexus3](https://nexus3.tumblr.net).
Then make sure to update our [Kafka-Connect deployment's Dockerfile](https://github.tumblr.net/TumblrAE/kafka-connect/blob/main/Dockerfile).
Run `mvn package [-DskipTests]` and upload the generated zip file to [Tumblr Nexus3](https://nexus3.tumblr.net).
Then make sure to update our [Kafka-Connect deployment's Dockerfile](https://github.tumblr.net/TumblrAE/kafka-connect/blob/main/Dockerfile)
with the new version tag.

```bash
export MATTICSPACE_USER=<your matticspace username>
export PLUGIN_VERSION=$( mvn org.apache.maven.plugins:maven-help-plugin:evaluate -Dexpression=project.version -q -DforceStdout)
# skip tests if you're _too_ confident
mvn package [-DskipTests]
curl -su "${MATTICSPACE_USER}" \
--upload-file "target/components/packages/confluentinc-kafka-connect-hdfs-${PLUGIN_VERSION}.zip" \
"https://nexus3.tumblr.net/nexus/repository/fs-private/sre/data/kafka-connect-plugins/confluentinc-kafka-connect-hdfs-${PLUGIN_VERSION}.zip" \
&& echo "Plugin uploaded" || echo "Plugin upload FAILED"
```

---

Expand Down

0 comments on commit 2b26ae1

Please sign in to comment.