Gh-370: Improve Federated Example

docker/gaffer-rest/README.md

@@ -1,88 +1,84 @@
-Gaffer REST
-===========
-In this folder you can find the required files for building and running Gaffer REST image for the REST API for Gaffer.
+# Gaffer REST
 
-# Configuration
-The image is based off Gaffer's spring-rest module. It runs a Gaffer REST API using a few different files.
+In this folder you can find the required files for building and running the Gaffer
+REST image for the REST API for Gaffer.
 
-### store.properties: 
-The store properties file tells Gaffer how to store its data. You have to provide a store class and store properties class. 
-To keep this image small, the only store that is supported is the MapStore which is an in-memory store. A default is provided
-in /gaffer/store/store.properties
+## Configuration
 
-### schema.json
-The Schema files make up a Gaffer schema which tells Gaffer what datasets are stored in your graph. It contains type information
-which a store may use to serialise and aggregate the data. You can put the whole schema in one file or split it up into as many
-as you choose. A basic schema is provided in /gaffer/schema by default
-
-### graphConfig.json
-The Graph config tells Gaffer what the graph is called as well as any hooks to run before an operation chain is run. A default
-is provided at /gaffer/graph/graphConfig.json
+The image is based off Gaffer's spring-rest module. It runs a Gaffer REST API
+using a few different files.
 
-### application.properties
-This is a spring concept and is used to change the context root and any properties related to Gaffer or the app. A default is
-provided at /gaffer/config/application.properties
-
-# Running Locally
-The easiest way to build and run these services is to use docker compose, by running the following from this directory:
-```bash
-docker compose up
-```
+### store.properties
 
-## Customising your builds
-We provide the options to build using an official release, a branch or a custom runnable rest.jar file
-The order it will check is:
+The store properties file tells Gaffer how to store its data. You have to
+provide a store class and store properties class. To keep this image small, the
+only store that is supported is the MapStore which is an in-memory store. A
+default is provided in `/gaffer/store/store.properties`
 
-1. Custom "rest.jar" file stored in "/jars"
-2. official release
-3. branch
+### schema.json
 
-To use a release or branch, set the GAFFER_VERSION property in env before running the `docker compose build` command.
+The Schema files make up a Gaffer schema which tells Gaffer what datasets are
+stored in your graph. It contains type information which a store may use to
+serialise and aggregate the data. You can put the whole schema in one file or
+split it up into as many as you choose. A basic schema is provided in
+`/gaffer/schema` by default
 
-We also provide the option to include custom libraries in the /jars/lib directory. These will be added to the classpath
-at runtime.
+### graphConfig.json
 
-## Containers that are started:
-* Gaffer REST
+The Graph config tells Gaffer what the graph is called as well as any hooks to
+run before an operation chain is run. A default is provided at
+`/gaffer/graph/graphConfig.json`
 
-Access the Gaffer REST API at: http://localhost:8080/rest/
+### application.properties
 
-# Proxy Example
+This is a spring concept and is used to change the context root and any
+properties related to Gaffer or the app. A default is provided at
+`/gaffer/config/application.properties`
 
-A demo/example of using Gaffer REST with a Proxy Store is available under the `proxy-example` directory.
+## Building/Running Locally
 
-## Running
-To run it from the `proxy-example` directory you'll need to specify the environment file path:
+The easiest way to build and/or run the REST API image is to use docker
+compose. You can use the following to just build the image:
 
 ```bash
-docker compose --env-file ../.env up
+docker compose build
 ```
 
-Or to run it from this directory, the compose file path and environment file path:
+Or to build and run the new image with default configuration run:
 
 ```bash
-docker compose -f proxy-example/compose.yaml --env-file .env up
+docker compose up
 ```
 
-With this deployment both a standard Gaffer REST API and a Gaffer REST API using the Proxy Store will be started.
-Only the port for the Proxy REST endpoint is exposed, this will forward operations to the standard Gaffer REST Endpoint.
+### Customising your builds
 
-# Federated Example
+We provide the options to build using an official release, a branch or a custom
+runnable `rest.jar` file. The order it will check is:
 
-A demo/example of using Gaffer REST with a Federated Store is available under the `federated-example` directory.
+1. Custom `rest.jar` file stored in `/jars`
+2. official release
+3. branch
 
-## Running
-To run it from the `federated-example` directory you'll need to specify the environment file path:
+To use a release or branch, set the `GAFFER_VERSION` property in env before
+running the `docker compose build` command.
 
-```bash
-docker compose --env-file ../.env up
-```
+We also provide the option to include custom libraries in the `/jars/lib`
+directory. These will be added to the classpath at runtime.
 
-Or to run it from this directory, the compose file path and environment file path:
+### Containers that are started
 
-```bash
-docker compose -f federated-example/compose.yaml --env-file .env up
-```
+* Gaffer REST
+
+Access the Gaffer REST API at: `http://localhost:8080/rest/`
+
+## Examples
+
+### Proxy Example
+
+A demo/example of using Gaffer REST with a Proxy Store is available under the
+`proxy-example` directory.
+
+### Federation Example
 
-This deploys a Gaffer REST API using the Federated Store. No graphs exist by default, they can be added using the API.
-The config doesn't specify a cache type, so a HashMap cache is used by default - changes do not persist with this cache.
+A demo/example of using Gaffer REST with a Federated Store is available under
+the `federated-example` directory.

docker/gaffer-rest/federated-example/README.md

@@ -0,0 +1,28 @@
+# Gaffer REST Federated Example
+
+This is an example deployment of Gaffer's federation feature. This uses a
+REST API deployment of Gaffer to which two in memory sub graphs are added
+which can be queried across.
+
+The example uses a split version of the
+[Tinkerpop Modern Graph](https://tinkerpop.apache.org/docs/current/tutorials/the-gremlin-console/#toy-graphs)
+where all the `knows` edges are in one graph with ID: `knowsGraph` and all
+the `created` and `software` vertexes are in another with graph ID: `createdGraph`.
+This leaves the `people` vertexes common across both graphs so that when
+federated it will appear as the original Modern graph data.
+
+## Running the Example
+
+To run the example please use the provided start script with an environment file
+to specify the image versions e.g. `accumulo2.env`:
+
+```bash
+./create-deployment.sh ../../accumulo2.env
+```
+
+With the demo deployment a standard Gaffer REST API will be started (available
+on port 8080 by default). The deployment uses map stores to store two sub graphs
+that can be federated across. Basic configuration for federation has been added
+(under the `config` directory) but can be modified to test out different
+scenarios, please see the [documentation](https://gchq.github.io/gaffer-doc/latest/administration-guide/gaffer-stores/federated-store.html)
+for more information.

docker/gaffer-rest/federated-example/compose.yaml

@@ -20,5 +20,7 @@ services:
     ports:
     - 8080:8080
     volumes:
-    - ./config/store.properties:/gaffer/store/store.properties:ro
-    - ./config/mergeFunctions.json:/gaffer/store/mergeFunctions.json:ro
+    - ./config/store/store.properties:/gaffer/store/store.properties:ro
+    - ./config/store/mergeFunctions.json:/gaffer/store/mergeFunctions.json:ro
+    - ./config/store/defaultGraphIds.json:/gaffer/store/defaultGraphIds.json:ro
+    - ./config/schema:/gaffer/schema:ro

docker/gaffer-rest/federated-example/config/schema/elements.json

@@ -0,0 +1,36 @@
+{
+    "entities": {
+        "person": {
+            "vertex": "string.id",
+            "properties": {
+                "name": "property.string",
+                "age": "property.integer"
+            }
+        },
+        "software": {
+            "vertex": "string.id",
+            "properties": {
+                "name": "property.string",
+                "lang": "property.string"
+            }
+        }
+    },
+    "edges": {
+        "knows": {
+            "source": "string.id",
+            "destination": "string.id",
+            "directed": "true",
+            "properties": {
+                "weight": "property.double"
+            }
+        },
+        "created": {
+            "source": "string.id",
+            "destination": "string.id",
+            "directed": "true",
+            "properties": {
+                "weight": "property.double"
+            }
+        }
+    }
+}

docker/gaffer-rest/federated-example/config/schema/types.json

@@ -0,0 +1,28 @@
+{
+    "types": {
+        "string.id": {
+            "class": "java.lang.String"
+        },
+        "true": {
+            "class": "java.lang.Boolean"
+        },
+        "property.string": {
+            "class": "java.lang.String",
+            "aggregateFunction": {
+                "class": "uk.gov.gchq.koryphe.impl.binaryoperator.First"
+            }
+        },
+        "property.integer": {
+            "class": "java.lang.Integer",
+            "aggregateFunction": {
+                "class": "uk.gov.gchq.koryphe.impl.binaryoperator.First"
+            }
+        },
+        "property.double": {
+            "class": "java.lang.Double",
+            "aggregateFunction": {
+                "class": "uk.gov.gchq.koryphe.impl.binaryoperator.First"
+            }
+        }
+    }
+}

docker/gaffer-rest/federated-example/config/store/defaultGraphIds.json

@@ -0,0 +1,4 @@
+[
+    "knowsGraph",
+    "createdGraph"
+]

docker/gaffer-rest/federated-example/config/store.properties → docker/gaffer-rest/federated-example/config/store/store.properties

@@ -14,3 +14,4 @@
 gaffer.store.properties.class=uk.gov.gchq.gaffer.federatedstore.FederatedStoreProperties
 gaffer.store.class=uk.gov.gchq.gaffer.federatedstore.FederatedStore
 gaffer.federatedstore.storeConfiguredMergeFunctions=/gaffer/store/mergeFunctions.json
+gaffer.federatedstore.storeConfiguredGraphIds=/gaffer/store/defaultGraphIds.json

docker/gaffer-rest/federated-example/create-deployment.sh

@@ -0,0 +1,63 @@
+#!/bin/sh
+# Copyright 2024 Crown Copyright
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+CURRENT_DIR="$(dirname "$(readlink -f "${0}")")"
+GAFFER_REST_URL="http://localhost:8080"
+KNOWS_GRAPH_JSON="${CURRENT_DIR}/operations/add-modern-knows-graph.json"
+CREATED_GRAPH_JSON="${CURRENT_DIR}/operations/add-modern-created-graph.json"
+
+# Make sure we have an evironment file for the image versions
+if [ ! -f "${1}" ]; then
+    echo "Error - Environment file not set"
+    echo "Example: ${0} <envfile>.env"
+    exit 1
+fi
+
+# Remove any existing containers
+docker compose --env-file "${1}" -f "${CURRENT_DIR}/compose.yaml" down
+
+# Start new deployment
+echo "-----------------------------------"
+echo "Starting new deployment..."
+docker compose --env-file "${1}" -f "${CURRENT_DIR}/compose.yaml" up --detach
+
+# Check can connect to instance
+echo "-----------------------------------"
+echo "Attempting to connect to Gaffer instance..."
+if ! docker run --rm --network=host curlimages/curl:latest \
+            curl --retry 20 \
+                 --retry-delay 5 \
+                 --retry-all-errors "${GAFFER_REST_URL}"/rest/graph/status
+then
+    echo "Failed to connect to Gaffer instance"
+    exit 1
+fi
+
+# Load graphs and data
+echo ""
+echo "-----------------------------------"
+echo "Creating 'knowsGraph' sub graph and loading data..."
+docker run --rm --network=host curlimages/curl:latest \
+       curl -H "Content-Type: application/json" \
+            --data "$(cat "${KNOWS_GRAPH_JSON}")" "${GAFFER_REST_URL}"/rest/graph/operations/execute
+echo "-----------------------------------"
+echo "Creating 'createdGraph' sub graph and loading data..."
+docker run --rm --network=host curlimages/curl:latest \
+       curl -H "Content-Type: application/json" \
+            --data "$(cat "${CREATED_GRAPH_JSON}")" "${GAFFER_REST_URL}"/rest/graph/operations/execute
+
+echo "-----------------------------------"
+echo "Deployment started!"
+echo "REST API availabe at: ${GAFFER_REST_URL}/rest"