Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Issue 3275][pulsar-io]Support source and sink of flume #3597

Merged
merged 24 commits into from Mar 4, 2019

Conversation

@tuteng
Copy link
Member

tuteng commented Feb 14, 2019

Motivation

#3275

Modifications

  • Support data of flume to pulsar
  • Support data of pulsar to flume

Document

flow of data

4

sink command
pulsar/bin/pulsar-admin sink localrun --archive pulsar/pulsar-io/flume/target/pulsar-io-flume-2.3.0-SNAPSHOT.nar --classname org.apache.pulsar.io.flume.sink.StringSink --tenant public --namespace default --name flume  --inputs flume_test --sink-config-file pulsar/pulsar-io/flume/src/main/resources/flume/flume-io-source.yaml --parallelism 1
source command
pulsar/bin/pulsar-admin source localrun --archive pulsar/pulsar-io/flume/target/pulsar-io-flume-2.3.0-SNAPSHOT.nar --classname org.apache.pulsar.io.flume.source.StringSource --tenant public --namespace default --name flume  --destination-topic-name flume_test --source-config-file pulsar/pulsar-io/flume/src/main/resources/flume/flume-io-sink.yaml --parallelism 1
other document

In editing

@aahmed-se

This comment has been minimized.

Copy link
Contributor

aahmed-se commented Feb 14, 2019

Flume has it's source sink system, not sure where we should integrate the source and sink

@tuteng

This comment has been minimized.

Copy link
Member Author

tuteng commented Feb 15, 2019

Flume has it's source sink system, not sure where we should integrate the source and sink

@aahmed-se That's true. This question has been discussed in issue #3275.

@aahmed-se

This comment has been minimized.

Copy link
Contributor

aahmed-se commented Feb 15, 2019

Sounds good.

@merlimat merlimat added this to the 2.4.0 milestone Feb 15, 2019
@tuteng

This comment has been minimized.

Copy link
Member Author

tuteng commented Feb 22, 2019

rerun java8 tests

1 similar comment
@tuteng

This comment has been minimized.

Copy link
Member Author

tuteng commented Feb 22, 2019

rerun java8 tests

@tuteng

This comment has been minimized.

Copy link
Member Author

tuteng commented Feb 22, 2019

run java8 tests
run integration tests

@tuteng

This comment has been minimized.

Copy link
Member Author

tuteng commented Feb 22, 2019

rerun java8 tests

@jiazhai jiazhai merged commit 37a2fda into apache:master Mar 4, 2019
3 checks passed
3 checks passed
Jenkins: C++ / Python Tests SUCCESS
Details
Jenkins: Integration Tests SUCCESS
Details
Jenkins: Java 8 - Unit Tests SUCCESS
Details
@merlimat

This comment has been minimized.

Copy link
Contributor

merlimat commented Apr 17, 2019

@tuteng I've seen a failure in the tests for this connector.

Standard Output
Created annotation listener
14:22:32.466 [main:org.apache.flume.source.AvroSource@205] ERROR org.apache.flume.source.AvroSource - Avro source null startup failed. Cannot initialize Netty server
org.jboss.netty.channel.ChannelException: Failed to bind to: /0.0.0.0:44444
	at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272) ~[netty-3.10.1.Final.jar:?]
	at org.apache.avro.ipc.NettyServer.<init>(NettyServer.java:106) ~[avro-ipc-1.8.1.jar:1.8.1]
	at org.apache.flume.source.AvroSource.start(AvroSource.java:202) [flume-ng-core-1.9.0.jar:1.9.0]
	at org.apache.pulsar.io.flume.sink.StringSinkTests.setUp(StringSinkTests.java:82) [test-classes/:?]
	....
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413) [surefire-booter-2.21.0.jar:2.21.0]
Caused by: java.net.BindException: Address already in use

I was trying to see if it's possible to check for an open port and use that from the config files.

We have already a class to get that in:

 <dependency>
   <groupId>${project.groupId}</groupId>
   <artifactId>managed-ledger-original</artifactId>
   <version>${project.version}</version>
   <scope>test</scope>
</dependency>

The you could just call PortManager.nextFreePort().

@codelipenghui

This comment has been minimized.

Copy link
Contributor

codelipenghui commented Jun 24, 2019

@tuteng Can you add some documents for the flume IO connector?

@tuteng

This comment has been minimized.

Copy link
Member Author

tuteng commented Jun 24, 2019

@tuteng Can you add some documents for the flume IO connector?

Ok, I will add it later.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.