Skip to content

Commit

Permalink
Merge pull request #323 from itesla/histodb_server
Browse files Browse the repository at this point in the history
Histo DB server:
Add some information in the readme file, and a small server management script to start/stop the server.
  • Loading branch information
sylvlecl committed Jan 12, 2018
2 parents ef947f2 + ddf2001 commit 50fb7cf
Show file tree
Hide file tree
Showing 4 changed files with 205 additions and 13 deletions.
7 changes: 7 additions & 0 deletions distribution/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,13 @@
<version>${project.version}</version>
</dependency>

<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>histodb-server</artifactId>
<version>${project.version}</version>
<classifier>exec</classifier>
</dependency>

<!-- Sampler (WP4.1) -->
<dependency>
<groupId>${project.groupId}</groupId>
Expand Down
8 changes: 4 additions & 4 deletions histodb-server/pom.xml
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
<?xml version="1.0" encoding="UTF-8"?>
<!-- Copyright (c) 2016, All partners of the iTesla project (http://www.itesla-project.eu/consortium)
This Source Code Form is subject to the terms of the Mozilla Public License,
v. 2.0. If a copy of the MPL was not distributed with this file, You can
obtain one at http://mozilla.org/MPL/2.0/. -->
<!-- Copyright (c) 2017, RTE (http://www.rte-france.com) This Source Code
Form is subject to the terms of the Mozilla Public License, v. 2.0. If a
copy of the MPL was not distributed with this file, You can obtain one at
http://mozilla.org/MPL/2.0/. -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
Expand Down
110 changes: 101 additions & 9 deletions histodb-server/readme.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,11 @@
# HistoDB server

## Installation
Put the histodb-server-<VERSION>-exec.jar in the designated folder.

Provide a valid SSL certificate (see key* configuration parameters)

## Configuration
Add a "histodb-server" section to the platform configuration setting the following parameters :

- persistent : true/false
- basedir : path to mapdb storage folder
- basedir : path to mapdb storage folder
- host : server address
- port : listening port
- username : user login
Expand All @@ -19,11 +15,107 @@ Add a "histodb-server" section to the platform configuration setting the followi
- keyStorePassword : keystore password
- keyAlias: key alias
- separator : csv separator char
- locale : locale (i.e. fr-FR)
- locale : locale (i.e. fr-FR)



## Startup/Shutdown
To start/stop the server use the "histodb-service.sh" script in platform bin folder:

histodb-service.sh {start|stop|restart|status}

## Service interface
The Histodb server exposes an HTTP interface to request data loading and to query historical data.

### Authentication
The server requires basic HTTP authentication with username and password.

### Load data
The histodb loads networks data from a specified local path on the histodb server host.
The load command can be given making an HTTP POST request to the URL

https://<host>:<port>/histodb/rest/itesladb/<datasource_prefix>/<datasource_postfix>/itesla

setting the following parameter:

- dir : local folder containing the network files to load

Example using wget command:

wget --user=username --password password https://<host>:<port>/histodb/rest/itesladb/<datasource_prefix>/<datasource_postfix>/itesla --post-data='dir=/caserepo/IIDM'

The load of many network files can take a log time to complete.
If using wget command be careful to specify a congruous timeout to avoid wget's default behavior of retrying the same request up to 20 times every 900 seconds (default timeout).
For example:

wget --timeout 200000 --user=username --password password https://<host>:<port>/histodb/rest/itesladb/<datasource_prefix>/<datasource_postfix>/itesla/ --post-data='dir=/caserepo/test/IIDM'

To avoid self-signed SSL certificate validation failures eventually add the "--no-check-certificate" wget option.

### Set reference network
When loading a set of network data (snapshots and forecasts) the last loaded snapshot will be kept as reference for the list of network attributes.
However it is possible to load a new reference network making a HTTP POST or PUT to the following URL:

/histodb/rest/itesladb/<datasource_prefix>/<datasource_postfix>/itesla/referenceCIM

setting the following parameter:

- dir : path to the snapshot network file

### Get Reference Network
To retrieve the current reference networkId make an HTTP GET the following URL:

/histodb/rest/itesladb/<datasource_prefix>/<datasource_postfix>/itesla/referenceCIM

## Query
To query historical data it is possible to make an HTTP GET request passing the query parameters on the query string, e.g.:

/histodb/rest/itesladb/<datasource_prefix>/<datasource_postfix>/itesla/data.<format>?<param1>=<value1>&<param2>=<value2>..


### Response format
Data can be formatted either as CSV file or as compressed CSV file specifying the requested format in the request URL as follows:

/histodb/rest/itesladb/<datasource_prefix>/<datasource_postfix>/itesla/data.<format>?<param1>=<value1>&<param2>=<value2>..

where format=csv|zip

/histodb/rest/itesladb/<datasource_prefix>/<datasource_postfix>/itesla/data.csv?<param1>=<value1>&<param2>=<value2>..

/histodb/rest/itesladb/<datasource_prefix>/<datasource_postfix>/itesla/data.zip?<param1>=<value1>&<param2>=<value2>..


### Get historical data
Historical data can be queried using the following URL :

/histodb/rest/itesladb/<datasource_prefix>/<datasource_postfix>/itesla/data.<format>?<param1>=<value1>&<param2>=<value2>...

### Get forecast diff
To retrieve forecasts with matching snapshot use the following URL:

/histodb/rest/itesladb/<datasource_prefix>/<datasource_postfix>/itesla/data/forecastsDiff.<format>?<param1>=<value1>&<param2>=<value2>...

### Get statistics
To get statistics use the following URL:

/histodb/rest/itesladb/<datasource_prefix>/<datasource_postfix>/itesla/stats.<format>?<param1>=<value1>&<param2>=<value2>...


## Startup
to start the server execute the following command:
### Query parameters
The list of possible query parameters contains:

java -jar histodb-server-<VERSION>-exec.jar
- headers: whether the result should contain column headers
- horizon: time horizon of the request (file type) : SN, ...
- count: count of data to be returned (default=50)
- start:
- cols:
- colRange: range of columns, as "min-max" or "*"
- forecast: forecast time
- time: [isodate,isodate]
- daytime: [isotime,isotime]
- powerType: comma separated list
- equip: comma separated list
- attr: comma separated list
- region: comma separated list
- country: comma separated list
- ids: comma separated list
93 changes: 93 additions & 0 deletions scripts/histodb-service.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
#!/bin/bash
#
# Copyright (c) 2017-2018, RTE (http://www.rte-france.com) This Source Code
# Form is subject to the terms of the Mozilla Public License, v. 2.0. If a
# copy of the MPL was not distributed with this file, You can obtain one at
# http://mozilla.org/MPL/2.0/.

installBinDir=$(dirname $(readlink -f $0))
installDir=${installBinDir%/*}

. $installDir/etc/itools.conf

pidFile=$HOME/.histodb_pid

TIME=`date +%Y-%m-%d_%H_%M_%S`

[ -n "$itools_cache_dir" ] && options+="-Ditools.cache.dir=$itools_cache_dir"
[ -n "$itools_config_dir" ] && options+=" -Ditools.config.dir=$itools_config_dir"
[ -n "$itools_config_name" ] && options+=" -Ditools.config.name=$itools_config_name"
[ -z "$java_xmx" ] && java_xmx=32G


usage() {
echo "`basename $0` {start|stop|restart|status}"
}

readPid() {
pid=""
if [ -f $pidFile ]; then
pid=`cat $pidFile`
# check pid exists anf if not clean pid file
kill -0 $pid > /dev/null 2>&1
if [ $? -ne 0 ]; then
rm -f $pidFile
fi
fi
}

start() {
readPid
if [ -n "$pid" ]; then
echo "histodb service already started ($pid)"
else
logsDir=$installDir/logs
mkdir $logsDir >> /dev/null 2>&1
mv $logsDir/histodb.log $logsDir/histodb.log_${TIME} >> /dev/null 2>&1
echo "starting histodb server ( execution details and errors will be logged in file: "$logsDir"/histodb.log )"
nohup $JAVA_HOME/bin/java -Xmx$java_xmx $options -cp $installDir/share/java -jar $installDir/share/java/histodb-server-0.1-SNAPSHOT-exec.jar > $logsDir/histodb.log 2>&1&
pid=$!
[ $? -eq 0 ] && echo $pid > $pidFile
sleep 5
grep -q " ERROR " $logsDir/histodb.log && echo "errors starting the online service. Please check the log file for details: "$logsDir"/histodb.log" || head -5 $logsDir/histodb.log
fi
}

stop() {
readPid
if [ -n "$pid" ]; then
echo "stopping histodb service"
kill $pid
rm -f $pidFile
fi
}

status() {
readPid
if [ -n "$pid" ]; then
echo "histodb service is running ($pid)"
else
echo "histodb service is not running"
fi
}



case "$1" in
"start")
start
;;
"stop")
stop
;;
"restart")
stop
start
;;
"status")
status
;;
*)
usage
;;
esac

0 comments on commit 50fb7cf

Please sign in to comment.