Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[pulsar-io] Add a Pulsar IO connector for Redis sink #3700

Merged
merged 10 commits into from
Mar 12, 2019

Conversation

murong00
Copy link
Contributor

Motivation

This PR provides a built-in Redis sink Connector, in order to cache messages in Redis [key-value] pairs. This will effectively make Redis a caching system, which other applications can access to get the latest value.

Modifications

Add a new sub-module in the pulsar-io module.

Verifying this change

This change can be verified as follows:

  • deploy the redis sink connector with configuration file containing the following fields:
configs:
    redisHosts: "localhost:6379"
    redisPassword: "redis@123"
    redisDatabase: "1"
    clientMode: "Standalone"
    operationTimeout: "3000"
    batchSize: "100"
  • start a redis instance with auth
  • send messages with NotNull key/value in the topic declared when deploying the connector
  • check in Redis if the message's key-value pairs have been stored in above database

Documentation

# Submit a Redis Sink
$ bin/pulsar-admin sink create --tenant public --namespace default --name redis-test-sink --sink-type redis --sink-config-file examples/redis-sink.yaml --inputs test_redis
    
# List Sink
$ bin/pulsar-admin sink list --tenant public --namespace default
    
# Get Sink Info
$ bin/pulsar-admin sink get --tenant public --namespace default --name redis-test-sink
    
# Get Sink Status
$ bin/pulsar-admin sink status --tenant public --namespace default --name redis-test-sink
    
# Delete the Redis Sink
$ bin/pulsar-admin sink delete --tenant public --namespace default --name redis-test-sink

@jiazhai
Copy link
Member

jiazhai commented Feb 28, 2019

run java8 tests

1 similar comment
@murong00
Copy link
Contributor Author

murong00 commented Mar 1, 2019

run java8 tests

@sijie
Copy link
Member

sijie commented Mar 2, 2019

run integration tests

@murong00
Copy link
Contributor Author

murong00 commented Mar 4, 2019

run java8 tests
run integration tests

@murong00
Copy link
Contributor Author

murong00 commented Mar 6, 2019

run java8 tests

1 similar comment
@murong00
Copy link
Contributor Author

murong00 commented Mar 7, 2019

run java8 tests

@murong00
Copy link
Contributor Author

run java8 tests

@murong00
Copy link
Contributor Author

run cpp tests
run java8 tests

@sijie sijie added type/feature The PR added a new feature or issue requested a new feature area/connector labels Mar 12, 2019
@sijie sijie added this to the 2.4.0 milestone Mar 12, 2019
@sijie sijie merged commit c5e3baa into apache:master Mar 12, 2019
@murong00 murong00 deleted the branch-3700 branch March 8, 2020 04:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/connector type/feature The PR added a new feature or issue requested a new feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants