Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add simple client for RabbitMQ #147

Closed
wants to merge 1 commit into from

Conversation

@Zlopez
Copy link

Zlopez commented Oct 9, 2019

This client is based on RabbitMQ Java
Client
.

Supports SSL and Username/Password authentication and could be
configured to work with any RabbitMQ server.

Connection to each server is created only once and separate channel is
created for each consumer and publisher.

When publishing generates message id using java.util.UUID and send this
id as property of the message.

Consuming is done using by using Push API (subscription) even when
listening for messages for a limited time.

Signed-off-by: Michal Konečný mkonecny@redhat.com

This client is based on [RabbitMQ Java
Client](https://www.rabbitmq.com/java-client.html).

Supports SSL and Username/Password authentication and could be
configured to work with any RabbitMQ server.

Connection to each server is created only once and separate channel is
created for each consumer and publisher.

When publishing generates message id using java.util.UUID and send this
id as property of the message.

Consuming is done using by using Push API (subscription) even when
listening for messages for a limited time.

Signed-off-by: Michal Konečný <mkonecny@redhat.com>
@Zlopez

This comment has been minimized.

Copy link
Author

Zlopez commented Oct 9, 2019

This implements #132

@pypingou

This comment has been minimized.

Copy link

pypingou commented Oct 28, 2019

Pretty ping? :)

@scoheb

This comment has been minimized.

Copy link
Contributor

scoheb commented Oct 29, 2019

@pypingou @Zlopez

I am getting back to reviewing PRs and the CI environment is having issues. I am testing things locally and will test this soon.

On a side note, how does this differ from #145 ?

@Zlopez

This comment has been minimized.

Copy link
Author

Zlopez commented Oct 29, 2019

@scoheb This is implemented using RabbitMQ client and supports everything RabbitMQ has to offer.

I used AMQP and Fedmsg as inspiration, but RabbitMQ has more features than these two.

@scoheb

This comment has been minimized.

Copy link
Contributor

scoheb commented Oct 29, 2019

@Zlopez so can your provider be used to connect to an AMQP instance/broker?

@Zlopez

This comment has been minimized.

Copy link
Author

Zlopez commented Oct 30, 2019

@scoheb I don't think so, the connection is established by using RabbitMQ ConnectionFactory.

@scoheb

This comment has been minimized.

Copy link
Contributor

scoheb commented Oct 30, 2019

I would like to see some basic tests that exercise sending and receiving messages. Please base yourself on the existing FedMsg and ActiveMq tests that make use of a Docker image to run rabbitmq.

@Zlopez

This comment has been minimized.

Copy link
Author

Zlopez commented Oct 30, 2019

I will look at the existing tests, but I wasn't able to run them when I wanted to test them.

@scoheb

This comment has been minimized.

Copy link
Contributor

scoheb commented Oct 30, 2019

I will be merging some fixes to make the tests more robust soon...

@Zlopez

This comment has been minimized.

Copy link
Author

Zlopez commented Oct 31, 2019

@scoheb Do you plan to merge to rabbitmq_skeleton branch too?

@Zlopez

This comment has been minimized.

Copy link
Author

Zlopez commented Oct 31, 2019

And one important question, how should I run those tests?

@scoheb

This comment has been minimized.

Copy link
Contributor

scoheb commented Oct 31, 2019

@Zlopez do you want to re-base off of master now instead of the skeleton branch?

As for running the tests, you need:

  • latest master of jms-messaging (due to jenkins core and plugins required)
  • docker (or podman)
  • firefox
  • geckodriver available in your path

then run:

  • mvn clean install -DskipTests
  • mvn -B test -DElasticTime.factor=2 -Djenkins.version=2.176.1 -DforkCount=1 -Dtest=FedMsgMessagingPluginIntegrationTest#testSimpleCIEventTriggerWithCheck

or remove the -Dtest= to run whole suite.

Warning they do take a while to run.

@Zlopez

This comment has been minimized.

Copy link
Author

Zlopez commented Oct 31, 2019

I could rebase to master, but I will create a new PR, I'm not sure if you could change existing PR to merge to different branch.

@pypingou

This comment has been minimized.

Copy link

pypingou commented Oct 31, 2019

@Zlopez if you edit the PR at the top, you can change the target branch :)

@Zlopez

This comment has been minimized.

Copy link
Author

Zlopez commented Oct 31, 2019

I already created a new branch in my fork for this, because the rebase to master branch didn't look well. So I will close this one, when I open a new one, which is correctly created from master branch.

@Zlopez

This comment has been minimized.

Copy link
Author

Zlopez commented Nov 1, 2019

@scoheb
I'm trying to run tests now, I have geckodriver in PATH, firefox installed and podman installed. But almost every test is skipped:

[INFO] Running com.redhat.jenkins.plugins.ci.MigrationTest                                                                                                                                                                                    
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.272 s - in com.redhat.jenkins.plugins.ci.MigrationTest                                                                                                              
[INFO] Running com.redhat.jenkins.plugins.ci.integration.AmqMessagingPluginLockdownIntegrationTest                                                                                                                                            
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 12.38 s - in com.redhat.jenkins.plugins.ci.integration.AmqMessagingPluginLockdownIntegrationTest                                                                    
[INFO] Running com.redhat.jenkins.plugins.ci.integration.FedMsgMessagingPluginIntegrationTest                                                                                                                                                 
[WARNING] Tests run: 30, Failures: 0, Errors: 0, Skipped: 30, Time elapsed: 252.076 s - in com.redhat.jenkins.plugins.ci.integration.FedMsgMessagingPluginIntegrationTest                                                                     
[INFO] Running com.redhat.jenkins.plugins.ci.integration.AmqMessagingPluginWithFailoverIntegrationTest                                                                                                                                        
[WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 50.945 s - in com.redhat.jenkins.plugins.ci.integration.AmqMessagingPluginWithFailoverIntegrationTest                                                               
@scoheb

This comment has been minimized.

Copy link
Contributor

scoheb commented Nov 1, 2019

It may be that warning message from podman screwing things up when it tries to detect if Docker is running or available...what does 'docker ps' give you? If it mentions to create a file to remove a message, I think that is the problem.

@Zlopez

This comment has been minimized.

Copy link
Author

Zlopez commented Nov 1, 2019

You were right, I didn't had docker installed and the docker ps wasn't recognized. Now I have it, but I'm getting this:

Skipping testSimpleCIEventTriggerWithCheck(com.redhat.jenkins.plugins.ci.integration.FedMsgMessagingPluginIntegrationTest)
org.junit.internal.AssumptionViolatedException: Docker is needed for the test
	at org.jenkinsci.test.acceptance.junit.WithDocker$RuleImpl$1.hasDocker(WithDocker.java:57)
	at org.jenkinsci.test.acceptance.junit.WithDocker$RuleImpl$1.evaluate(WithDocker.java:49)
	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
	at org.jenkinsci.test.acceptance.junit.JenkinsAcceptanceTestRule$1.evaluate(JenkinsAcceptanceTestRule.java:58)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
1572611919126	Marionette	INFO	Stopped listening on port 35709

What I tried:

  • set alias docker=podman
  • systemctl restart docker

And still I'm getting the error above.

@scoheb

This comment has been minimized.

Copy link
Contributor

scoheb commented Nov 1, 2019

in my env, podman is installed...not docker.
The check in the tests simply does:

docker ps and expects a '0' exit code.

Here is the result from my local env:

$ which docker
/usr/bin/docker

$ file /usr/bin/docker
/usr/bin/docker: a /usr/bin/sh script, ASCII text executable

$ cat  /usr/bin/docker
#!/usr/bin/sh
[ -f /etc/containers/nodocker ] || \
echo "Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg." >&2
exec /usr/bin/podman "$@"

$ docker ps
CONTAINER ID  IMAGE  COMMAND  CREATED  STATUS  PORTS  NAMES

$ echo $?
0

@Zlopez

This comment has been minimized.

Copy link
Author

Zlopez commented Nov 1, 2019

Thanks, the bash script did the trick.

@Andrei-Stepanov

This comment has been minimized.

Copy link

Andrei-Stepanov commented Nov 4, 2019

Hi, we have a plan to deploy the plugin on https://jenkins-continuous-infra.apps.ci.centos.org/
The master already has jms-messaging-plugin.
Question: can the updated plugin continue work with fedora-fedmsg?

@Zlopez

This comment has been minimized.

Copy link
Author

Zlopez commented Nov 4, 2019

@Andrei-Stepanov Fedmsg is different messaging system, so it is possible to use this beside RabbitMQ.

@Zlopez

This comment has been minimized.

Copy link
Author

Zlopez commented Nov 5, 2019

@scoheb I was able to run the tests without docker error, I'm now trying to run mvn -B test -DElasticTime.factor=2 -Djenkins.version=2.176.1 -DforkCount=1 -Dtest=FedMsgMessagingPluginIntegrationTest#testSimpleCIEventTriggerWithCheck

But it's failing with this:

Building Docker image `docker build -t jenkins/fedmsg-relay:e048987d8a07 /tmp/Dockerfile5088748677418623093dir`: logfile is at target/diagnostics/testSimpleCIEventTriggerWithCheck(com.redhat.jenkins.plugins.ci.integration.FedMsgMessagingPluginIntegrationTest)/docker-FedmsgRelayContainer.build.log
Launching Docker container `docker run -d -p 127.0.0.1::4001 -p 127.0.0.1::2003 -p 127.0.0.1::22 jenkins/fedmsg-relay:e048987d8a07`: logfile will be at target/diagnostics/testSimpleCIEventTriggerWithCheck(com.redhat.jenkins.plugins.ci.integration.FedMsgMessagingPluginIntegrationTest)/docker-FedmsgRelayContainer.run.log
Lis 01, 2019 4:25:03 ODP. org.jenkinsci.test.acceptance.update_center.MockUpdateCenter lambda$serverExceptionHandler$5
WARNING: Exception thrown while serving request
java.net.SocketException: Connection reset
	at java.net.SocketInputStream.read(SocketInputStream.java:210)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
	at org.apache.http.impl.io.DefaultHttpRequestParser.parseHead(DefaultHttpRequestParser.java:129)
	at org.apache.http.impl.io.DefaultHttpRequestParser.parseHead(DefaultHttpRequestParser.java:53)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
	at org.apache.http.impl.DefaultBHttpServerConnection.receiveRequestHeader(DefaultBHttpServerConnection.java:129)
	at org.apache.http.protocol.HttpService.handleRequest(HttpService.java:308)
	at org.apache.http.impl.bootstrap.Worker.run(Worker.java:66)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

127.0.0.1:36807
127.0.0.1:41561
master53872|Nov 01, 2019 4:25:38 PM com.redhat.jenkins.plugins.ci.CIBuildTrigger stopTriggerThreads
master53872|INFO: Removed thread lock for 'representative_glow'.
master53872|Nov 01, 2019 4:25:38 PM com.redhat.jenkins.plugins.ci.CIBuildTrigger startTriggerThreads
master53872|INFO: Starting thread (183) for 'representative_glow'.
master53872|Nov 01, 2019 4:25:38 PM com.redhat.jenkins.plugins.ci.ProjectChangeListener onUpdated
master53872|INFO: Getting trigger threads.
master53872|Nov 01, 2019 4:25:39 PM com.redhat.jenkins.plugins.ci.messaging.FedMsgMessagingWorker subscribe
master53872|INFO: Subscribing job 'representative_glow' to org.fedoraproject topic.
master53872|Nov 01, 2019 4:25:39 PM com.redhat.jenkins.plugins.ci.messaging.FedMsgMessagingWorker subscribe
master53872|INFO: Successfully subscribed job 'representative_glow' to topic 'org.fedoraproject'.
master53872|Nov 01, 2019 4:25:53 PM com.redhat.jenkins.plugins.ci.ProjectChangeListener onUpdated
master53872|INFO: No CIBuildTrigger found, forcing thread stop.
master53872|Nov 01, 2019 4:25:53 PM com.redhat.jenkins.plugins.ci.CIBuildTrigger stopTriggerThreads
master53872|INFO: Removed thread lock for 'trivial_liquid'.
master53872|Nov 01, 2019 4:25:56 PM hudson.model.Run execute
master53872|INFO: trivial_liquid #1 main build action completed: SUCCESS
master53872|Nov 01, 2019 4:25:56 PM com.redhat.utils.MessageUtils sendMessage
master53872|INFO: Sending message for job 'trivial_liquid'.
master53872|Nov 01, 2019 4:26:03 PM com.redhat.jenkins.plugins.ci.messaging.JMSMessagingWorker trigger
master53872|INFO: Scheduling job 'representative_glow' based on message:
master53872|{"content":"catch me"}
master53872|Nov 01, 2019 4:26:03 PM hudson.model.Run execute
master53872|INFO: representative_glow #1 main build action completed: SUCCESS
1572621965303	Marionette	INFO	Stopped listening on port 46111

###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
@Zlopez

This comment has been minimized.

Copy link
Author

Zlopez commented Nov 11, 2019

Closing this in favor of #154

@Zlopez Zlopez closed this Nov 11, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.