Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FCREPO-3753 #166

Merged
merged 36 commits into from Oct 15, 2021
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
0636a82
added http service
Sep 23, 2021
43ab524
Fix comment.
demiankatz Oct 4, 2021
842d311
Remove unnecessary dependency.
demiankatz Oct 4, 2021
44e71fe
Ensures the fedora id is sent in the body on reindex.
dbernstein Oct 4, 2021
f5a2931
Adds missing mustache file.
dbernstein Oct 5, 2021
d883035
Adds in mustache dep.
dbernstein Oct 5, 2021
794a624
Merge pull request #1 from dbernstein/http-service-fix
demiankatz Oct 5, 2021
b3930d3
Finishing touches.
demiankatz Oct 5, 2021
11aae05
Improve comment.
demiankatz Oct 5, 2021
6e98016
Use constant; add initial test.
demiankatz Oct 6, 2021
a0828b3
more progress
Oct 6, 2021
6f37847
Fix broken tests.
demiankatz Oct 6, 2021
251d949
readme updates
Oct 7, 2021
8c8b6d7
Merge branch 'FCREPO-3753' of https://github.com/Geoffsc/fcrepo-camel…
Oct 7, 2021
aa57713
Merge branch 'main' into FCREPO-3753
demiankatz Oct 7, 2021
3fcee5a
README updates
Oct 7, 2021
5b10fe0
Incomplete, non-functional integration tests (WIP).
demiankatz Oct 7, 2021
a2540b8
Progress on tests.
demiankatz Oct 7, 2021
849dd95
Merge remote-tracking branch 'upstream/main' into FCREPO-3753
demiankatz Oct 11, 2021
a800275
Test style improvement.
demiankatz Oct 11, 2021
c5b4d04
Log error for missing HTTP base URL.
demiankatz Oct 11, 2021
17bd46c
Update MockServer + add WireMock
mikejritter Oct 11, 2021
38c2b6d
Create a resource in fedora before testing
mikejritter Oct 11, 2021
f821cc6
Fail fast
mikejritter Oct 11, 2021
9084326
Switch to wiremock; remove unneeded endpoints
mikejritter Oct 11, 2021
ec45ce0
Merge pull request #2 from mikejritter/FCREPO-3753
demiankatz Oct 12, 2021
5d18b81
Test progress.
demiankatz Oct 12, 2021
373ea19
Send appropriate headers for tests.
demiankatz Oct 12, 2021
9584d77
Clean up; remove unnecessary Fedora logic.
demiankatz Oct 12, 2021
fc153d5
Remove unnecessary Jetty/Fedora logic.
demiankatz Oct 12, 2021
dee9860
Refactor README.
demiankatz Oct 12, 2021
347f95e
Add note about required property.
demiankatz Oct 12, 2021
fc916b8
Merge remote-tracking branch 'upstream/main' into FCREPO-3753
demiankatz Oct 12, 2021
a7c09f3
Simplify code.
demiankatz Oct 12, 2021
c870401
Checkstyles.
demiankatz Oct 12, 2021
8bf3c15
Merge branch 'main' into FCREPO-3753
demiankatz Oct 15, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
6 changes: 6 additions & 0 deletions fcrepo-camel-toolbox-app/pom.xml
Expand Up @@ -32,6 +32,12 @@
<version>${project.parent.version}</version>
</dependency>

<dependency>
<groupId>${project.parent.groupId}</groupId>
<artifactId>fcrepo-indexing-http</artifactId>
<version>${project.parent.version}</version>
</dependency>

<dependency>
<groupId>${project.parent.groupId}</groupId>
<artifactId>fcrepo-reindexing</artifactId>
Expand Down
110 changes: 110 additions & 0 deletions fcrepo-indexing-http/pom.xml
@@ -0,0 +1,110 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

<modelVersion>4.0.0</modelVersion>

<parent>
<artifactId>fcrepo-camel-toolbox</artifactId>
<groupId>org.fcrepo.camel</groupId>
<version>6.0.0-SNAPSHOT</version>
</parent>

<artifactId>fcrepo-indexing-http</artifactId>
<packaging>jar</packaging>

<name>Fedora Indexing Service with a Http backend</name>
<description>Camel-based indexing service for Http</description>

<properties>
</properties>

<dependencies>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-core</artifactId>
</dependency>

<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-spring-javaconfig</artifactId>
</dependency>

<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-http</artifactId>
</dependency>

<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-mustache</artifactId>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect that some of these dependencies aren't actually needed here; we based this pom.xml on the Solr component, and there's probably some junk that can be trimmed out.

</dependency>

<dependency>
<groupId>org.fcrepo.camel</groupId>
<artifactId>fcrepo-camel</artifactId>
</dependency>

<dependency>
<groupId>org.fcrepo.client</groupId>
<artifactId>fcrepo-java-client</artifactId>
</dependency>

<dependency>
<groupId>${project.parent.groupId}</groupId>
<artifactId>fcrepo-camel-common</artifactId>
<version>${project.parent.version}</version>
</dependency>

<!-- logging -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
</dependency>

<!-- Testing & Camel Plugin -->
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-test-spring</artifactId>
</dependency>

<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>xerces</groupId>
<artifactId>xercesImpl</artifactId>
</dependency>

</dependencies>

<build>
<defaultGoal>install</defaultGoal>

<plugins>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
</plugin>

<!--plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
</plugin-->

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-resources-plugin</artifactId>
</plugin>

<!-- to generate the MANIFEST-FILE of the bundle -->
</plugins>
</build>
</project>
@@ -0,0 +1,85 @@
/*
* Licensed to DuraSpace under one or more contributor license agreements.
* See the NOTICE file distributed with this work for additional information
* regarding copyright ownership.
*
* DuraSpace licenses this file to you under the Apache License,
* Version 2.0 (the "License"); you may not use this file except in
* compliance with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.fcrepo.camel.indexing.http;

import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.component.http.HttpComponent;
import org.fcrepo.camel.common.config.BasePropsConfig;
import org.fcrepo.camel.common.config.ConditionOnPropertyTrue;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Conditional;
import org.springframework.context.annotation.Configuration;

/**
* A configuration class for the Http Indexer service
*
* @author Geoff Scholl, Demian Katz
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the convention for @author tags? Is this okay, or should there be a separate line for each author?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Separate line I believe is preferable.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks -- fixed!

*/
@Configuration
@Conditional(FcrepoHttpIndexerConfig.HttpIndexerEnabled.class)
public class FcrepoHttpIndexerConfig extends BasePropsConfig {

static final String HTTP_INDEXER_ENABLED = "http.indexer.enabled";

static class HttpIndexerEnabled extends ConditionOnPropertyTrue {
HttpIndexerEnabled() {
super(FcrepoHttpIndexerConfig.HTTP_INDEXER_ENABLED, false);
}
}

@Value("${input.stream:broker:topic:fedora}")
dbernstein marked this conversation as resolved.
Show resolved Hide resolved
private String inputStream;

@Value("${http.reindex.stream:broker:queue:http.reindex}")
private String reindexStream;

@Value("${filter.containers:http://localhost:8080/fcrepo/rest/audit}")
dbernstein marked this conversation as resolved.
Show resolved Hide resolved
private String filterContainers;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure what this is for; is it used?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is for ignoring messages from resources that start with the specified paths.


@Value("${http.baseUrl:http://localhost:8983/http/collection1}")
dbernstein marked this conversation as resolved.
Show resolved Hide resolved
private String httpBaseUrl;

public String getInputStream() {
return inputStream;
}

public String getReindexStream() {
return reindexStream;
}

public String getHttpBaseUrl() {
return httpBaseUrl;
}


@Bean(name = "http")
public HttpComponent http() {
return new HttpComponent();
}

@Bean(name = "https")
public HttpComponent https() {
return new HttpComponent();
}

@Bean
public RouteBuilder httpRoute() {
return new HttpRouter();
}
}
@@ -0,0 +1,82 @@
/*
* Licensed to DuraSpace under one or more contributor license agreements.
* See the NOTICE file distributed with this work for additional information
* regarding copyright ownership.
*
* DuraSpace licenses this file to you under the Apache License,
* Version 2.0 (the "License"); you may not use this file except in
* compliance with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.fcrepo.camel.indexing.http;

import org.apache.camel.LoggingLevel;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.support.builder.Namespaces;
import org.fcrepo.camel.processor.EventProcessor;
import org.slf4j.Logger;
import org.springframework.beans.factory.annotation.Autowired;

import static org.apache.camel.Exchange.HTTP_METHOD;
import static org.slf4j.LoggerFactory.getLogger;

/**
* A content router for handling JMS events.
*
* @author Geoff Scholl, Demian Katz
*/
public class HttpRouter extends RouteBuilder {

private static final Logger logger = getLogger(HttpRouter.class);

@Autowired
private FcrepoHttpIndexerConfig config;

/**
* Configure the message route workflow.
*/
public void configure() throws Exception {

final Namespaces ns = new Namespaces("rdf", "http://www.w3.org/1999/02/22-rdf-syntax-ns#");
ns.add("indexing", "http://fedora.info/definitions/v4/indexing#");
ns.add("ldp", "http://www.w3.org/ns/ldp#");

/*
* A generic error handler (specific to this RouteBuilder)
*/
onException(Exception.class)
.maximumRedeliveries(config.getMaxRedeliveries())
.log("Index Routing Error: ${routeId}");

/*
* route a message to the proper queue
*/
from(config.getInputStream())
.routeId("FcrepoHttpRouter")
.process(new EventProcessor())
.log(LoggingLevel.TRACE, "Received message from Fedora routing to index.http")
.to("direct:send.to.http");

/*
* Handle re-index events
*/
from(config.getReindexStream())
.routeId("FcrepoHttpReindex")
.to("direct:send.to.http");
dbernstein marked this conversation as resolved.
Show resolved Hide resolved

/*
* Forward message to Http
*/
from("direct:send.to.http").routeId("FcrepoHttpSend")
.log(LoggingLevel.INFO, logger, "sending to http...")
.setHeader(HTTP_METHOD).constant("POST")
.to(config.getHttpBaseUrl());
}
}
1 change: 1 addition & 0 deletions pom.xml
Expand Up @@ -83,6 +83,7 @@
<module>fcrepo-indexing-triplestore</module>
<module>fcrepo-reindexing</module>
<module>fcrepo-indexing-solr</module>
<module>fcrepo-indexing-http</module>
<module>fcrepo-service-activemq</module>
<module>fcrepo-service-camel</module>
<module>fcrepo-ldpath</module>
Expand Down