The data exchange API provides API services for the exchange web UI (future), the edge nodes, and agreement Bots.
The exchange service also provides a few key services for BH for areas in which the decentralized P2P tools do not scale well enough yet. As soon as the decentralized tools are sufficient, they will replace these services in the exchange.
- Install scala
- Install sbt
- (optional) Install conscript and giter8 if you want to get example code from scalatra.org
- Install postgresql locally (unless you have a remote instance you are using). Instructions for installing on Mac OS X:
brew install postgresqlecho 'host all all <my-local-subnet> trust' >> /usr/local/var/postgres/pg_hba.confsed -i -e "s/#listen_addresses = 'localhost'/listen_addresses = 'my-ip'/" /usr/local/var/postgres/postgresql.confbrew services start postgresql- test:
psql "host=<my-ip> dbname=postgres user=<myuser> password=''"
- Add a config file on your development system at /etc/horizon/exchange/config.json with at least the following content (this is needed for the automated tests). Defaults and the full list of config variables are in
src/main/resources/config.json:
{
"api": {
"db": {
"jdbcUrl": "jdbc:postgresql://localhost/postgres", // my local postgres db
"user": "myuser",
"password": ""
},
"smtp": {
"host": "mysmtp.relay.com", // the SMTP relay svr the exchange uses to send pw reset emails
"user": "myemail@email.net", // email address
"password": "myemailpw" // email pw
},
"logging": {
"level": "DEBUG"
},
"root": {
"password": "myrootpw",
"email": ""
}
}
}
- Set the same exchange root password in your shell environment, for example:
export EXCHANGE_ROOTPW=myrootpw
- If someone hasn't done it already, create the TLS private key and certificate:
export EXCHANGE_KEY_PW=<pass-phrase>
make gen-key
- Otherwise, get files
exchangecert.pem,keypassword, andkeystorefrom the person who created them and put them in./keys/etc.
sbtjetty:start- Or to have the server restart automatically when code changes:
~;jetty:stop;jetty:start - Once the server starts, to try a simple rest method browse: http://localhost:8080/v1/admin/version
- To see the swagger output, browse: http://localhost:8080/api
- (Optional) To include tests for IBM agbot ACLs:
export EXCHANGE_AGBOTAUTH=myibmagbot:abcdef - (Optional) To include tests for IBM IAM platform key authentication:
export EXCHANGE_IAM_KEY=myiamplatformkey
export EXCHANGE_IAM_EMAIL=myaccountemail@something.com
export EXCHANGE_IAM_ACCOUNT=myibmcloudaccountid
- Run the automated tests in a second shell (with the exchange server still running in the first):
sbt test - Run just 1 of the the automated test suites (with the exchange server still running):
sbt "testOnly exchangeapi.AgbotsSuite" - Run the performance tests:
src/test/bash/scale/test.shorsrc/test/bash/scale/wrapper.sh 8
- Update the version in
src/main/resources.version.txt - To build the build container, compile your local code, build the exchange container, and run it, run:
make. Or you can do the individual steps:- Build the build container:
make .docker-bld - Build the code from your local exchange repo in the build container:
make .docker-compile - Build the exchange api container and run it locally:
make .docker-exec-run
- Build the build container:
- Manually test container locally:
curl -sS -w %{http_code} http://localhost:8080/v1/admin/version- Note: the container can not access a postgres db running locally on the docker host if the db is only listening for unix domain sockets or 127.0.0.1.
- Run the automated tests:
sbt test - Push container to the docker hub registry:
make docker-push-only - Deploy the new container to a docker host
- Ensure that no changes are needed to the /etc/horizon/exchange/config.json file
- Test the new container :
curl -sS -w %{http_code} https://<exchange-host>/v1/admin/version - To see the swagger info from the container:
https://<exchange-host-and-port>/api - Log output of the exchange svr can be seen via
docker logs -f exchange-api, or might also go to/var/log/syslogdepending on the docker and syslog configuration. - At this point you probably want to
make cleanto stop your local docker container so it stops listening on your 8080 port, or you will be very confused when you go back to running new code in your sandbox, and your testing doesn't seem to be executing it.
- Granular (per org) service ACL support:
- add Access type of BROWSE that will allow to see these fields of services:
- label, description, public, documentation, url, version, arch
- add acl table and resource with fields:
- org
- resource (e.g. service)
- resourceList (for future use)
- requester (org/username)
- access
- add GET, PUT, POST to manage these acls
- add checks on GET service to use these acls
- (later) consider adding a GET that returns all services of orgs of orgType=IBM
- add Access type of BROWSE that will allow to see these fields of services:
- Add rest method to delete a user's stale devices (carl requested)
- Add ability to change owner of node
- Add patch capability for node registered services
- Consider:
- detect if pattern contains 2 services that depend on the same exclusive MS
- detect if a pattern is updated with service that has userInput w/o default values, and give warning
- If maxAgreements>1, for CS, in search don't return node to agbot if agbot from same org already has agreement for same service.
- Consider changing all creates to POST, and update (via put/patch) return codes to 200
- Configure https/ssl at run time (instead of build time), so the same container can run with both http and https, or just http.
- Increase db auth access timeout to avoid timeouts under certain conditions.
- Add TLS/SSL support to jetty, instead of relying on the front end (e.g. haproxy) to terminate the SSL
- Increase default max number of resources by an order of magnitude
- Update scale test driver
- Support composite service url in
POST /orgs/{orgid}/patterns/{pattern}/search - Support composite service url in
/orgs/{orgid}/search/nodes
- add field to org called orgType, which will have value IBM for any of our orgs that have ibm services in them
- allow any user to GET all orgs with filter of orgType=IBM
- delete support for the special
publicorg - fix bug in which in the access denied error msg it only reported the generic access needed
- POST /orgs/{orgid}/nodes/{id}/configstate to POST /orgs/{orgid}/nodes/{id}/services_configstate
- In node resource, make msgEndPoint and softwareVersions optional
- Add node post to be able to be able to update some of the configState attrs
- Remove Resource from swagger and Services
- Support authentication via ibm cloud iam token that comes from the web ui
- Fix support for multiple orgs associate with 1 ibm cloud account
- Make orgs/ibmcloud_id table column not need to be unique, to support multiple orgs for the same ibm cloud account
- Fix problem with spaces in creds (issue 90)
- Handled special case of getting your own user when using iamapikey
- Added better errors for ibm auth
- Added /admin/clearAuthCaches
- Added IBM Cloud auth automated tests
- Added IBM Cloud auth plugin
- Fixed swagger
- Cleaned up README.md
- Improved error in
POST/PUT/PATCH patternwhen reference service does not exist to output service org, url, arch, version range - Added optional
documentationfield to service resource - Added new
resourcedefinition that can be required by services to hold things like models - Added
singletonas a validsharablevalue - Added JAAS authentication framework
- Remove microservice, workload, and blockchain support (leaving just service support)
- Remove deprecated
PUT /orgs/{orgid}/agbots/{id}/patterns/{patid}method (use POST instead)
- Support IAM API keys for bx cr access by adding
usernamefield to service dockauths (issue anax 651)
- Fixed creds in url parms when root creds are specified or when /orgs is queried (exchange-api issue 77)
- Fixed creds in url parms from 1 org querying a resource in another org (exchange-api issue 58)
- Add nodeOrgids to body of
/org/{orgid}/patterns/{pat-id}/nodehealth
- Make some more fields in POST/PUT patterns and services optional.
- Support wildcard pattern in
POST /orgs/{orgid}/agbots/{id}/patternsbody
PUT /orgs/{orgid}/agbots/{id}/patterns/{patid}: add nodeOrgid to the bodyPOST /orgs/{orgid}/patterns/{pattern}/search: add a list of nodeOrgid to the body
- Automatically run /admin/initdb or /admin/upgradedb on startup
- Added
agreementLessattribute toservicessection of pattern
- If POST /services/{svcid}/dockerauths is a dup, just update the current one
- Moved docker image to docker hub openhorizon
- Add services sub-resource called dockerauths for docker image auth tokens
- Fixed error upgrading db schema from earlier than db schema 3
- Fix build error in Schema.scala in wiotp enviroment
- allow IBM agbots to access private ms/wl/svc/pat via get /workloads etc.
- explicitly set http response code of 200 for successful GETs to avoid occasional 404's
- clean up variables in nodes suite tests
- add back in a few tests for workloads and microservices in pattern and node search
- pattern node search api now validates the serviceUrl
- change agbot agreement to service
- change put node status to service
- change name of service package field to imageStore
- test built image with compose db and verify exchange version output
- change node search to service
- change node, node agreement, pattern search
- changed pattern to support service
- Agbot agreement resources still refer to workloadOrgid, workloadPattern, workloadUrl (but you can put service info in those fields, because they are not checked)
- Node status resources still refer to workloads and microservices (but i think you can put service info in those fields, because i don't think they are checked)
- added keys to ms, wk, service, pattern (anax issue 498)
- Figured out how to set the Response codes in swagger
- Fixed the swagger 2 level resource problem
- fixed some library versions to make the build work again
- added service resource (issue #51)
- detect if svc requires other svc with different arch
- Allowed missing or empty torrent field in microservice and workload
- Made swagger work again, with these limitations:
- "Try it out" button still doesn't work
- The longer "description" field of each method is not appearing
- Added dbSchemaVersion field to the output of /admin/status
- Upgraded:
- scala from 2.12.1 to 2.12.4
- scalatra from 2.5.1 to 2.6.2
- from scalatra-sbt 0.5.1 to sbt-scalatra 1.0.1
- Allowed node in other org to access agbot in IBM org
- When creating/updating a workload and verifying the existence of the apiSpecs, it now treats the MS versions as ranges
- (No need to upgrade the db if coming from version 1.38 or later, altho it won't hurt either)
- Added GET /admin/version
- Docker tag for exchange docker image no longer has the "v" before the version number
- For the references between resources that are not enforced by DB foreign keys, added checking when the resource is created, updated, or patched:
- microservices referenced by a workload
- workloads referenced by a pattern
- patterns referenced by an agbot
- pattern referenced by a node
- Made the following resource fields optional:
- pattern: dataVerification can be omitted or specified as
{} - microservice: downloadUrl can be omitted, matchHardware can be omitted or specified as
{} - workload: downloadUrl can be omitted
- pattern: dataVerification can be omitted or specified as
(No need to upgrade the db if coming from version 1.38 or later, altho it won't hurt either)
- Updated jetty version to 9.4.7 and fixed build to pull latest bug fixes in the 9.4 range
- Fixed non-pattern node search to not find pattern nodes
- Now filter both pattern and non-pattern node searches to not return nodes with empty publicKey values
- Added
"nodeHealth": { "missing_heartbeat_interval": 600, "check_agreement_status": 120 }policy to patterns (it is a peer to the dataVerification field). Existing pattern resources in the DB will be converted on the way out. For new POST/PUTs this new field is optional in 1.40. - Added POST /orgs/{orgid}/patterns/{patid}/nodehealth and POST /orgs/{orgid}/search/nodehealth for agbot to get node lastHeartbeat and agreement status
(No need to upgrade the db if coming from version 1.38, altho it won't hurt either)
- Added PUT/GET/DELETE /orgs/{id}/agbots/{id}/patterns/{id} and removed the patterns field from the agbot resource. This allows each org/pattern pair of an agbot to be managed individually.
- If you are coming from exchange versions 1.35 - 37, you can run POST /admin/upgradedb (which preserves the data). If coming from an earlier version, you must drop all of the tables and run POST /admin/initdb.
- Added PUT/GET/DELETE /orgs/{id}/nodes/{id}/status
- If you are coming from v1.35.0 or v1.36.0, you can run POST /admin/upgradedb (which preserves the data). If coming from an earlier version, you must drop all of the tables and run POST /admin/initdb.
- Allow queries in another org at the /microservices, /workloads, /patterns, /bctypes/{id}/blockchains level (and only return the public resources)
- Added 'public' as a filter attribute for microservices, workloads, patterns.
- Removed old/unused sections api.specRef, api.objStoreTmpls, and api.microservices from config.json
- Replaced empty {} return from PUT nodes/{nodeid} with normal {"code": 201, "msg": "node added or updated"}
- Supported config.json api.limits.* equal to 0 to mean unlimited
- Fixed bug in which users, agbots, and nodes couldn't read their own org
- Need to dropdb and initdb, and re-enter data
- Fixed bug: open-horizon#35
- Enabled IBM agbots to read all patterns, search for nodes with patterns in other orgs, and msg nodes in other orgs (and they can msg it back)
- Enabled identities in other orgs to read all public patterns/workloads/microservices/blockchains
- Fix bug: Put root/root in the db, so foreign keys work correctly
- Now automatically create a public org and allow anonymous to create user in that org
- Ensured that a user can't elevate himself to an admin user
- Added schema table to store the current db schema and changed /admin/upgradedb to use that to upgrade correctly from any version. In this version of the exchange you still need to dropdb/initdb, but for all subsequent versions you should be able to use /admin/upgradedb.
- Need to dropdb and initdb, and re-enter data
- Quick fix to not require a slash in the node pattern attribute if it is blank
- Need to dropdb and initdb, and re-enter data
- Added pattern field to node resource
- Move arch field 1 level up in pattern schema, and renamed it to workloadArch
- Implemented /org/{orgid}/patterns/{pat-id}/search
- Need to dropdb and initdb, and re-enter data
- Removed the option on PUT users/{username} to set password or email to blank to mean don't update it. Implemented PATCH users/{username} instead.
- Changed config.json acls READ_MY_ORGS and WRITE_MY_ORGS to their singular versions
- Added 'admin' field to user, enabled admin users to do everything in their org (including create other users)
- Updated the pattern resource schema to more explicitly support multiple different workloads, and multiple versions of the same workload (see swagger)
- Need to dropdb and initdb, and re-enter data
- Added deployment_overrides and deployment_overrides_signature to patterns.workloads
- Added check_rate (int) in the dataVerification
- Need to dropdb and initdb, and re-enter data
- Cross-org access only works with root for now
- Before nodes, agbots, microservices, workloads, or patterns are created via front end auth, you have to create the user (and then use that person-type auth) so the owner foreign key succeeds
- Users, nodes, and agbots can now read their own org resource
- Renamed devices to nodes
- Removed properties, counterPartyProperties, microservices, maxAgreements from pattern resource
- Nodes can now read their own /org/{orgid} resource
- Moved bctype and blockchain under /org/{orgid} and added public field
- Modified PUT orgs/{org}/nodes/{id}/agreements/{agreementid} to include the pattern and workload, and to accept "microservices":[{"url":"ms url","org":"myorg"}] instead of an array of url strings like it is now.
- Modified PUT orgs/{org}/agbots/{id}/agreements/{agreementid} to accept workload orgid, pattern, and url, instead of just the url
- Added list of orgs/patterns to agbot resource
- A new setting
"frontEndHeader": "issuer"is required in therootsection of config.json to use data power in front of exchange
- Tested dropdb with older compose db
- Modified tests to have each suite use its own org
- Added support for data power headers
- Need to dropdb and initdb, and re-enter data
- All resource except /admin are now under /org/{orgid}. By convention we will use "IBM" for our orgid.
- All identities must be prefixed by "{orgid}/". E.g. IBM/myuser, IBM/mydeviceid, IBM/myagbotid (as a special case, the root user is root/root). This includes identities listed in owner and definedBy fields.
- Added 'public' field to microservices and workloads
- Change returned http code to 404 when there are no resource found for GET /org/{orgid}/users, /org/{orgid}/devices, /org/{orgid}/agbots
- Anonymous POST user/{username} (to create user) and POST user/{username}/reset are no longer supported (because we can't let them do that in any org)
- This rare case is no longer supported: no id is specified in the credentials and it defaults to the device/agbot id in the url
- Added the /orgs/{orgid} resource
- Moved under /orgs/{orgid}: users, devices, agbots, microservices, workloads
- Updated ACL to only only access within your org (with a few exceptions)
- Added 'public' field to microservices and workloads
- Got automated tests to pass with pre-existing IBM org
- Added org resource (still need to implement ACLs for it and move other resources under it)
- Changed PUT /devices/{id}/agreements/{agid} to accept list of microservices (and corresponding GET and POST search changes)
- Fixed bug in which the CORS header Access-Control-Allow-Origin was not set in the response
- Added /microservices and /workloads resources
- Changed default permissions to allow devices and agbots to read all agbots
- Changed additional image tag from latest to volcanostaging
- Converted to the newer build.sbt file and upgraded to sbt version 0.13.13
- Migrated to version 9.4.x of jetty
- Made ACLs read from config.json (instead of hardcoded)
- Fixed bug: with a brand new db, could not run POST /admin/initdb because root user was not defined.
- Switched the dockerfile FROM statement to the official jetty image
- Added msgs, bctypes, and blockchains to the scale test script
- Removed requirement for MS numAgreements be set to 1.
- Added PUT/GET/DELETE /bctypes/{type} and PUT/GET/DELETE /bctypes/{type}/blockchains/{name} to store/retrieve BC info
- All DELETE methods not return 404 if the resource was not found. (Before it was inconsistent, some returned 204 and some 404.)
- Refactored authentication and authorization code, so now it is easier to read, easier to extend, and handles a few corner cases better. No external changes for all the mainline use cases.
- Changed the scala version used in building the container to be the latest stable and the same as my dev environment (2.12.1)
- Moved up to latest stable version of scalatra (2.4.1 ot 2.5.1)
- Access denied msgs now give details
- Fixed pw reset email to give link to the swagger entry to change your pw with the token
- Added DELETE /devices/{id}/agreements and DELETE /agbots/{id}/agreements to delete all of the agreements of a device/agbot
- Added GET /(devices|agbots)/{id}?attribute={attrname} to get a single attribute of the device/agbot resource
- Added PATCH /(devices|agbots)/{id} to set a single attribute of the device/agbot resource
- Modified PUT /devices/{id} and PUT /agbots/{id} to also take publicKey in the input body (but made it backward compatible so you do not have to specify it)
- Added POST devices/{device-id}/msgs (to send/create a msg), GET devices/{device-id}/msgs (to read your msgs), and DELETE devices/{device-id}/msgs/{msg-id}. Also added the same methods for agbots.
- Fixed POST /admin/dropdb to not delete the root user from the authentication cache, so you have credentials to run POST /admin/initdb
- Improved the implementation of POST /search/devices. All externals are the same, except now if all devices are stale it returns 404 (the json returned is still
{"devices":[],"lastIndex":0}) - Added GET /admin/status to use for monitoring the exchange api svr
- Added checks so users can not create more devices, agbots, agreements, or msgs than the max numbers allowed in config.json, and added the associated maxMessagesInMailbox in the default config.json
- Removed all code for the in-memory db (it was not being maintained anyway)
- Added logging a real client IP (gotten from haproxy)
- adjusted debug logging to be more helpful and less verbose
- fixed POST admin/migratedb
- added support for base64 encoded credentials in the header
- added optional filter params to GET /devices and /agbots
- now log who each rest api call is from
- root user is now saved in the persistent db (but still loaded/updated from config.json at startup)
- deleting a user will now delete all of its devices, agbots, and agreements (we now have a proper foreign key between users and devices/agbots, and we use the sql on delete cascade option, so now the db completely takes care of object referencial integrity)