Skip to content

EDI messages receiver for SaaS owners

License

Notifications You must be signed in to change notification settings

updcon/edi-receiver

 
 

Repository files navigation

EDI-Receiver

EDI-Receiver

Clojure CI

Docker Image PG CI Docker Image MySQL CI Docker PG Load Test

A clojure application designed to receive and store EDI messages from the TBT Cloud into a customer’s local database. The solution is intended to be quite universal so is actively used to integrate other platforms (like, for example, DKDHUB) and applications into the robust and flexible digital environment.

Latest release assets available here.

NOTE

Default configuration implies PG and rely on upstream versioning

Customisations

There are a lot of possible customisations available as well as adding a new schemas or changing the schema sources. On order to get more information, please use the contact form here

Makefile build

In order to build and run with Makefile you should first install docker, have Leiningen installed and resolve build dependencies.

Install dependencies

NOTE

Default JDK/JRE may vary between distros, but by default Java 11+ is intended

sudo apt-get install make default-jdk-headless

Run standalone

Standalone run means you have external PostgreSQL server and successfully prepared your configuration

## 1st time to compile
make compile
## 1st time to synchronize schemas and initialize tables
make run-sync
## normally just
make run

Run as a dock’ed bundle

PG
sudo make docker-run
MySQL
sudo make docker-run-mysql

Benchmark

sudo make docker-run-load-pg

Manual build

Running with docker

Install docker first, then run

PG
sudo docker build -t edi-receiver -f docker/Dockerfile.pg .
MySQL
sudo docker build -t edi-receiver -f docker/Dockerfile.mysql .
Run
sudo docker run -it edi-receiver

Installing clojure

sudo apt-get update
sudo apt-get install wget default-jdk
sudo wget https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein -P /bin
sudo chmod a+x /bin/lein
lein upgrade

Build the application

lein deps
lein build-all

Installing postgresql

sudo apt-get update
sudo apt-get install -y wget gnupg
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt/ `lsb_release -cs`-pgdg main" >> /etc/apt/sources.list.d/pgdg.list'
sudo apt-get update
sudo apt-get install postgresql

Creating empty PG database

su - postgres -c "psql --command \"ALTER USER postgres WITH PASSWORD 'postgres';\""
su - postgres -c "psql --command \"CREATE DATABASE edi;\""

Running the application

To view migration SQL:

java -jar target/edi-control-standalone.jar --print-deploy-sql

To initialize or migrate database:

java -jar target/edi-control-standalone.jar --deploy

To start receiver:

java -jar target/edi-receiver-standalone.jar

Config

Config is java .properties file:

parameter type description

upstream.topics

string

comma separated list of topics to receive

upstream.cache

string

path where upstream schemas cached

upstream.sync

boolean

set true to validate upstream cache on startup

api.host

string

http server ip address

api.port

integer

http server port, default is 8000

api.auth.username

string

basic auth username, if auth required

api.auth.password

string

basic auth password, if auth required

db

string

which storage to use, "pg" or "mysql". Options from jdbc.* will be merged with corresponding options

pg.host

string

postgresql host, default is "localhost" (from jdbc.host)

pg.port

integer

postgresql port, default is 5432

pg.database

string

postgresql database, default is "edi" (from jdbc.database)

pg.user

string

postgresql user

pg.password

string

postgresql password

mysql.host

string

mysql host, default is "localhost" (from jdbc.host)

mysql.port

integer

mysql port, default is 3306

mysql.database

string

mysql database, default is "edi" (from jdbc.database)

mysql.user

string

mysql user

mysql.password

string

mysql password

See edi-receiver.properties for defaults and more options.

Customizing config

cp resources/edi-receiver.properties local.properties

Then edit local.properties file and run

java -jar target/edi-receiver-standalone.jar -c local.properties

Default properties will be updated with local.properties

Some config options can be customized from command line, see help:

java -jar target/edi-receiver-standalone.jar --help

Proxying

Sample configuration for proxying

backend.0.name="dear-kafka"
backend.0.type="kafka"
backend.0.bootstrap-servers="127.0.0.1:9092"
backend.0.enabled=true

backend.1.name="weak-connection-api"
backend.1.type="http"
backend.1.method="post"
backend.1.uri="http://my.host/api/topic/{topic}" ; {topic} will be substituted with proxy.N.target
backend.1.auth.enabled=true
backend.1.auth.type="basic"
backend.1.auth.username="test"
backend.1.auth.password="pass"
backend.1.enabled=true

proxy.0.backend="dear-kafka"
proxy.0.source="document"
proxy.0.target="proxy_document" ; kafka topic name
proxy.0.enabled=true
proxy.0.reliable=true           ; means not to accept message if kafka producer fails

; optional buffer configuration, used only with reliable=true
proxy.0.buffer.enabled=true         ; when true, if producer fails, keep message in buffer
proxy.0.buffer.max-size=1000        ; maximum buffer size, will return 422 in case of buffer owerflow
proxy.0.buffer.max-tries=3          ; retries number to send message, then message will be dropped
proxy.0.buffer.expire-time-s=180    ; message expiration time, seconds, then message will be dropped, can be nil (= forever)
proxy.0.buffer.tries-interval-ms=10000     ; minimal retry interval, default is 10000

; optional logging configuration. Used to store backend responses in "log" table
proxy.0.logging.enabled=true
proxy.0.logging.reference-fields="doctype,id" ; selected field names, stored as reference

proxy.1.backend="weak-connection-api"
proxy.1.source="document"
proxy.1.target="proxy_document"
proxy.1.condition="(= sender \\"tbt\\")"
proxy.1.transform="[[body (str body) (some? body)]]"
proxy.1.enabled=true
proxy.1.reliable=false          ; means to ignore api fails

Logging backend responses

Backend responses can be stored into log table.

Column Postgresql type Mysql type Fescription

created_at

timestamptz

datetime

time record created at

context

jsonb

json

Proxy configuration, e.g.: {"key":0,"target":"proxy_document","backend":"dear-kafka"}

reference

jsonb

json

Message reference. Selected message keys, as defined in setting reference-fields.

content

jsonb

json

Backend response

raw

bytea

blob

Backend response raw content, HTTP backend only

То enable logging, add following lines to proxy configuration:

proxy.0.logging.enabled=true
proxy.0.logging.reference-fields="field1,field2"

Setting reference-fields is mandatory. log.reference will be stored like:

{"field1": "value1", "field2": "value2"}

Transform

Transformation rules:

[:restrictive                           ; optional modifier, makes rules restrictive
 [target.path <expression> <condition>]
 [another.path <expression2>]           ; condition can be optional
 ...]

Restrictive means to start result from empty map. For example, following rules

[:restrictive
 [a (str (+ aa 10)) (odd aa)]   ; aa is odd, so evaluates
 [a (str aa) (even aa)]         ; no hit as aa is not even
 [b bb]                         ; simple assignment
 [c cc (some? cc)]              ; assign c from cc if cc is not nil
 [new 777]]                     ; constant new value

applied to

{:aa 1
 :bb 2
 :cc nil}

will produce

{:a   "11"
 :b   2
 :new 777}

Non-restrictive rules can be used to update or drop some values, for example, rules

[[aa (str (+ aa 10)) (odd aa)]  ; aa is odd so update it
 [aa (str aa) (even aa)]        ; no hit as aa is not even
 [cc :dissoc]                   ; remove cc anyway
 [new 777]]                     ; constant new value

applied to

{:aa 1
 :bb 2
 :cc nil}

will produce

{:aa  "11"
 :bb  2                         ; there was no rule on bb
 :new 777}

Benchmarking

Warning: benchmarking will affect your database and backends!

To send test requests to receiver, run

java -jar target/edi-control-standalone.jar --fire fire-config.json

where fire-config.json is file with content like:

{
  "threads": 10,
  "requests-per-thread": 1000,
  "payloads": [
    {
      "uri": "http://localhost:8000/api/topic/document",
      "method": "post",
      "auth": {
        "enabled": true,
        "type": "basic",
        "username": "test",
        "password": "pass"
      },
      "headers": {
        "Content-Type": "application/json"
      },
      "body": "{\"sender\":\"tbt\",\"timestamp\":\"2004-10-19 10:23:54+02\",\"doctype\":\"DocReturnOrders\",\"id\":\"3d9cddc1-f71d-0d6a-6343-010ccfc800a6\",\"body\":{\"a\":\"b\"},\"checksum\":\"fd6b34c54a331ccbbfacade3ea4bd225\"}"
    }
  ]
}

edi-control will run 10 threads with 1000 requests per thread and calculate average request time and request count per second. If multiple payloads supplied, payload for each request will be selected randomly.

Development

Execute (go) in repl to start development profile with autoreload and local.properties config.

License

Copyright © 2020 Kasta Group LLC

Copyright © 2020-2022 TBT LLC

This program and the accompanying materials are made available under the terms of the Eclipse Public License 2.0 which is available at http://www.eclipse.org/legal/epl-2.0.

This Source Code may also be made available under the following Secondary Licenses when the conditions for such availability set forth in the Eclipse Public License, v. 2.0 are satisfied: GNU General Public License as published by the Free Software Foundation, either version 2 of the License, or (at your option) any later version, with the GNU Classpath Exception which is available at https://www.gnu.org/software/classpath/license.html.

Sponsor this project

 

Packages

No packages published

Languages

  • Clojure 97.4%
  • Other 2.6%