RYA-443 Rya Streams Query Manager daemon program. #272

kchilton2 · 2018-02-08T21:15:46Z

Description

Implemented an RPM installed CentOS 7 daemon service used to execute Kafka Streams jobs on the local machine. This is part of Rya Streams.

Tests

Wrote automated tests and executed the application on a CentOS 7 machine using the RPM.

Links

https://issues.apache.org/jira/browse/RYA-443

Updated QueryRepository to be a Service Updated InMemoryQueryRepository to be an AbstractScheduledService Added listeners to InMemoryQueryRepository

…onfiguration file.

… single QueryChangeLogSource.

QueryManager with tests updated InMemoryQueryRepository and its tests

…ation.

…anager application.

…OS 7.

asfgit · 2018-02-08T22:19:18Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/686/

Build result: FAILURE

[...truncated 3.04 MB...][INFO] Apache Rya Web Projects ............................ SKIPPED[INFO] Apache Rya Web Implementation ...................... SKIPPED[INFO] ------------------------------------------------------------------------[INFO] BUILD FAILURE[INFO] ------------------------------------------------------------------------[INFO] Total time: 38:10 min[INFO] Finished at: 2018-02-08T22:18:44+00:00[INFO] Final Memory: 301M/2452M[INFO] ------------------------------------------------------------------------Waiting for Jenkins to finish collecting data[ERROR] Failed to execute goal org.codehaus.mojo:rpm-maven-plugin:2.1.5:attached-rpm (create-rpm-distribution) on project rya.streams.query-manager: Unable to query for default vendor from RPM: Error while executing process. Cannot run program "rpm": error=2, No such file or directory -> [Help 1][ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.[ERROR] Re-run Maven using the -X switch to enable full debug logging.[ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles:[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException[ERROR] [ERROR] After correcting the problems, you can resume the build with the command[ERROR] mvn -rf :rya.streams.query-managerchannel stoppedSetting status of b889bc0 to FAILURE with url https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/686/ and message: 'FAILURE 'Using context: Jenkins: clean package -Pgeoindexing

asfgit · 2018-02-08T22:26:17Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/687/

Build result: FAILURE

[...truncated 3.01 MB...][INFO] Apache Rya Spark Support ........................... SKIPPED[INFO] Apache Rya Web Projects ............................ SKIPPED[INFO] Apache Rya Web Implementation ...................... SKIPPED[INFO] ------------------------------------------------------------------------[INFO] BUILD FAILURE[INFO] ------------------------------------------------------------------------[INFO] Total time: 40:57 min[INFO] Finished at: 2018-02-08T22:25:43+00:00[INFO] Final Memory: 546M/3106M[INFO] ------------------------------------------------------------------------[ERROR] Failed to execute goal org.codehaus.mojo:rpm-maven-plugin:2.1.5:attached-rpm (create-rpm-distribution) on project rya.streams.query-manager: Unable to query for default vendor from RPM: Error while executing process. Cannot run program "rpm": error=2, No such file or directory -> [Help 1][ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.[ERROR] Re-run Maven using the -X switch to enable full debug logging.[ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles:[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException[ERROR] [ERROR] After correcting the problems, you can resume the build with the command[ERROR] mvn -rf :rya.streams.query-managerchannel stoppedSetting status of 50a8121 to FAILURE with url https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/687/ and message: 'FAILURE 'Using context: Jenkins: clean package -Pgeoindexing

ejwhite922

Looks good. Mostly minor stuff.

ejwhite922 · 2018-02-09T13:47:37Z

extras/rya.streams/api/src/main/java/org/apache/rya/streams/api/entity/StreamsQuery.java

+        sb.append(getSparql() + "\n");
+        sb.append("Is ");
+        if (!isActive) {
+            sb.append(" Not ");


No space is needed before "Not". The previous append has one.

ejwhite922 · 2018-02-09T13:49:37Z

...ya.streams/api/src/main/java/org/apache/rya/streams/api/queries/InMemoryQueryRepository.java

+        } finally {
+            lock.unlock();
+        }
+        log.trace("runOneIteration() - Exit");


Should this log message be moved up into the finally block so it's called if an Exception is thrown?

ejwhite922 · 2018-02-09T13:54:48Z

extras/rya.streams/api/src/main/java/org/apache/rya/streams/api/queries/QueryChange.java

+               "    Query ID: " + queryId + ",\n" +
+               "    Change Type: " + changeType + ",\n" +
+               "    Is Active: " + isActive + ",\n" +
+               "    SPARQL: " + sparql + "\n" +


Is "sparql" coming from user input? Might want to wrap that in org.apache.rya.api.log.LogUtils.clean(sparql)

What does that function do?

I'm going to assume it is sanitized by the time it reaches this line of code.

ejwhite922 · 2018-02-09T14:02:59Z

...treams/api/src/test/java/org/apache/rya/streams/api/queries/InMemoryQueryRepositoryTest.java

                // Listing the queries should work using an initialized change log.
                final Set<StreamsQuery> stored = initializedQueries.list();
                assertEquals(expected, stored);
+            } finally {
+                queries.stop();


This seems redundant with the outer try-finally block calling stop(). Unless the intent is to test it handling 2 consecutive calls to stop(). The inner try-finally can probably be removed.

ejwhite922 · 2018-02-09T14:06:09Z

extras/rya.streams/kafka/src/main/java/org/apache/rya/streams/kafka/KafkaTopics.java

+    }
+
+    /**
+     * Get the Rya instance name from a Kafka topic name that has been used for a {@link QueryChnageLog}.


typo. QueryChangeLog

ejwhite922 · 2018-02-09T14:39:07Z

...nager/src/main/java/org/apache/rya/streams/querymanager/kafka/KafkaQueryChangeLogSource.java

+/**
+ * Represents a Kafka Server that manages {@link KafkaQueryChangeLog}s.
+ * <p/>
+ * Thread safe.


Change to @ThreadSafe

What's the dependency for that annotation?

net.jcip.annotations.ThreadSafe

Looks like Rya is using this one:

<dependency> <groupId>com.github.stephenc.jcip</groupId> <artifactId>jcip-annotations</artifactId> <version>1.0-1</version> </dependency>

ejwhite922 · 2018-02-09T14:58:39Z

.../rya.streams/query-manager/src/test/java/org/apache/rya/streams/querymanager/ThreadUtil.java

+/**
+ * Utilities that are useful for interacting with {@link Thread}s while testing.
+ */
+public class ThreadUtil {


Add a private constructor to prevent instantiation

ejwhite922 · 2018-02-09T15:10:11Z

...a.streams/client/src/main/java/org/apache/rya/streams/client/command/ListQueriesCommand.java

+            sb.append("ID: ").append(query.getQueryId())
+                .append("    ")
+                .append("Is Active: ").append(query.isActive())
+                .append( query.isActive() ? "     " : "    " )


Maybe change to:
.append("Is Active: ")
.append(StringUtils.rightPad(query.isActive(), 9))

ejwhite922 · 2018-02-09T15:12:01Z

extras/rya.streams/api/src/main/java/org/apache/rya/streams/api/entity/StreamsQuery.java

+        sb.append("ID: ");
+        sb.append(getQueryId().toString() + "\n");
+        sb.append("Query: ");
+        sb.append(getSparql() + "\n");


Is "getSparql()" coming from user input? Might want to wrap that in org.apache.rya.api.log.LogUtils.clean(getSparql())

Yes, but it should be sanitized before it gets here.

ejwhite922 · 2018-02-09T15:12:58Z

...a.streams/client/src/main/java/org/apache/rya/streams/client/command/ListQueriesCommand.java

+                .append("    ")
+                .append("Is Active: ").append(query.isActive())
+                .append( query.isActive() ? "     " : "    " )
+                .append("Query: ").append(query.getSparql()).append("\n");


LogUtils.clean(query.getSparql())

Should be sanitized before it gets here.

kchilton2 · 2018-02-11T02:34:33Z

How do we want to deal with this project producing an RPM? The ASF jenkins doesn't have the rpm command installed.

asfgit · 2018-02-14T20:56:27Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/691/

Build result: FAILURE

[...truncated 3.03 MB...][INFO] Apache Rya Spark Support ........................... SKIPPED[INFO] Apache Rya Web Projects ............................ SKIPPED[INFO] Apache Rya Web Implementation ...................... SKIPPED[INFO] ------------------------------------------------------------------------[INFO] BUILD FAILURE[INFO] ------------------------------------------------------------------------[INFO] Total time: 37:36 min[INFO] Finished at: 2018-02-14T20:56:07+00:00[INFO] Final Memory: 535M/2904M[INFO] ------------------------------------------------------------------------[ERROR] Failed to execute goal org.codehaus.mojo:rpm-maven-plugin:2.1.5:attached-rpm (create-rpm-distribution) on project rya.streams.query-manager: Unable to query for default vendor from RPM: Error while executing process. Cannot run program "rpm": error=2, No such file or directory -> [Help 1][ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.[ERROR] Re-run Maven using the -X switch to enable full debug logging.[ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles:[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException[ERROR] [ERROR] After correcting the problems, you can resume the build with the command[ERROR] mvn -rf :rya.streams.query-managerchannel stoppedSetting status of 6428ca8 to FAILURE with url https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/691/ and message: 'FAILURE 'Using context: Jenkins: clean package -Pgeoindexing

ejwhite922

Looks good.

kchilton2 · 2018-03-02T19:17:59Z

I'm closing this because #279 takes it over.

isper3at and others added 15 commits February 2, 2018 16:44

RYA-446 Daemonifying Query Manager

6688c2e

RYA-449 Create QueryChangeLogSource

8803d7d

Rya 454 added QueryExecutor interface

c048db7

Rya 452 Updated QueryRepository

0dad129

Updated QueryRepository to be a Service Updated InMemoryQueryRepository to be an AbstractScheduledService Added listeners to InMemoryQueryRepository

RYA-455 stopAll queries for a rya instance

3810fb7

RYA-448 Implement JAXB marshalling code for the Query Manager's XML c…

e774ef0

…onfiguration file.

RYA-450 Implemented a Kafka backed QueryChangeLogSource.

13cfd43

RYA-458 Updating the configuration XML so that you may only specify a…

eee8d83

… single QueryChangeLogSource.

RYA-456 Implement a Single Node implementation of QueryExecutor.

815d995

Rya 451 Query manager

ba2898a

QueryManager with tests updated InMemoryQueryRepository and its tests

RYA-453 Implement the Query Manager's Daemon that controls the applic…

34e616c

…ation.

RYA-446 Create a bin and rpm distribution for the Rya Streams Query M…

8d35b45

…anager application.

RYA-446 Service unit file for systemd

4f4e0f4

RYA-446 Making the Rya Streams Query Manager run as a service on Cent…

4a4456e

…OS 7.

RYA-451 Fixing threading issues with the QueryManager class.

50a8121

kchilton2 force-pushed the RYA-443_rya-streams-querymanager branch from b889bc0 to 50a8121 Compare February 8, 2018 21:30

ejwhite922 reviewed Feb 9, 2018

View reviewed changes

Code review.

6428ca8

ejwhite922 approved these changes Feb 22, 2018

View reviewed changes

isper3at approved these changes Feb 23, 2018

View reviewed changes

kchilton2 closed this Mar 2, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RYA-443 Rya Streams Query Manager daemon program. #272

RYA-443 Rya Streams Query Manager daemon program. #272

kchilton2 commented Feb 8, 2018

asfgit commented Feb 8, 2018

asfgit commented Feb 8, 2018

ejwhite922 left a comment

ejwhite922 Feb 9, 2018

kchilton2 Feb 14, 2018

ejwhite922 Feb 9, 2018

kchilton2 Feb 14, 2018

ejwhite922 Feb 9, 2018

kchilton2 Feb 9, 2018

kchilton2 Feb 14, 2018

ejwhite922 Feb 9, 2018

kchilton2 Feb 14, 2018

ejwhite922 Feb 9, 2018

kchilton2 Feb 14, 2018

ejwhite922 Feb 9, 2018

kchilton2 Feb 9, 2018

ejwhite922 Feb 22, 2018

ejwhite922 Feb 9, 2018

kchilton2 Feb 14, 2018

ejwhite922 Feb 9, 2018

kchilton2 Feb 14, 2018

ejwhite922 Feb 9, 2018

kchilton2 Feb 14, 2018 •

edited

Loading

ejwhite922 Feb 9, 2018

kchilton2 Feb 14, 2018 •

edited

Loading

kchilton2 commented Feb 11, 2018

asfgit commented Feb 14, 2018

ejwhite922 left a comment

kchilton2 commented Mar 2, 2018

RYA-443 Rya Streams Query Manager daemon program. #272

RYA-443 Rya Streams Query Manager daemon program. #272

Conversation

kchilton2 commented Feb 8, 2018

Description

Tests

Links

asfgit commented Feb 8, 2018

Build result: FAILURE

asfgit commented Feb 8, 2018

Build result: FAILURE

ejwhite922 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kchilton2 Feb 14, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kchilton2 Feb 14, 2018 • edited Loading

Choose a reason for hiding this comment

kchilton2 commented Feb 11, 2018

asfgit commented Feb 14, 2018

Build result: FAILURE

ejwhite922 left a comment

Choose a reason for hiding this comment

kchilton2 commented Mar 2, 2018

kchilton2 Feb 14, 2018 •

edited

Loading

kchilton2 Feb 14, 2018 •

edited

Loading