Skip to content
An application can create and manage data lineage information, by using IOTA tangle
TypeScript HTML JavaScript CSS
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.vscode
DataLineageApp
Doc
.deployment
.gitattributes
.gitignore
.gitmodules
.travis.yml
DataLineageApp.sln
LICENSE
README.md
TestAccess.txt

README.md

Data Lineage Service Build Status

This data lineage viewer is a web based application that visualize the data lineage information, based on the paper Data Integrity and Lineage by using IOTA.

It also provides user-friendly web interface for publishing integrity and lineage information to tangle.

In addition, all functionalities are exposed as APIs, which helps developers to build/extend their applications to integrate with tangle quickly, without having deep understanding of IOTA or MAM protocol.

Live demo

https://datalineage-viewer.azurewebsites.net

The demo application is connecting to the IOTA Mainnet, by using the following node(s):

NOTE: Public nodes can be offline without notice. Check more public nodes from community, such as https://www.tangle-nodes.com/index.php?sorts[load]=1

Functionalities

Lineage viewer

This application can visualize the data lineage information, such as input/output, owner information, etc.

Publish data to tangle

This application also provide user friendly interface for publishing data to tangle. Since these simulators do talk to IOTA mainnet, you can use them to write into tangle that will be available for all iota users.

  1. Publish data integrity information
  2. Publish data lineage information

APIs

all functionalities are exposed as APIs. The detailed API documentation can be found at https://datalineage-viewer.azurewebsites.net/swagger/

Technical details

Architecture

Architecture

This application is using the official IOTA MAM library https://github.com/iotaledger/mam.client.js for interacting with IOTA tangle. Besides, it is based on Nodejs and Typescript for both frontend and server side.

Connect to IOTA network

The application supports fall-back mechanism for connecting to multiple nodes: You specify a list of nodes in config file. The application will try to use the first one. If that is not working, then the application try to use the second one, etc... It helps especially if you are using public nodes and not sure if they are online.

It also supports proxy to speed up the access to the IOTA.

Outsourcing PoW

In order to speed up MAM message throughput, instead of run PoW on the Iota node that the application is connecting with, this task is delegated to 3rd party Pow provider https://powsrv.io/ (with free tier).

Override Configuration

As mentationed in above, you can specify the IOTA node address (and other settings) in config file, but we also support to overwrite the value in the static source code. When server start running, it will load the content of file server-config first (it is a json file that is of the same schema of the IConfig) in the data folder. Then, if one of the config values is not presented in the loaded file content, the default value from the source code will be used. So it is possible to provide the server-config file to overwrite the default value in source code.

Caching

In order to have a good performance, also considering the data is immutable in tangle (assuming it talks to a permanode), we decide to use caching.

It means that, whenever the client is requesting package information from an address, the API will check the memory cache first.

  • if the data for this address already exists in cache, simply return it.
  • if the data does not exist, the server will talk to IOTA node that it is connecting with, fetch the data, store it in cache, and then return it. For every 3 minutes, the server will save the memory cache to disk file. In case the server was reboot, the cache can be loaded from file to memory.

The cache files are stored in /data folder of the web application server.

  • package.cache
  • iotaWriter.cache (one iotawriter object per one seed/channel, all stored in this cache file)

Application Insight

This app use the Microsoft Application Insight to tracking and logging. If the application is hosted in Azure, you can provide an ikey via the environment variable APPINSIGHTS_INSTRUMENTATIONKEY, or set it directly in the code logger.ts, for more information, please refer to the document Monitor your Node.js services and apps with Application Insights

How to init submodule after cloned the source code

git submodule update --init 

How to build and run locally

To build the code, in folder "/DataLineageApp/", run the following commands:

npm install
npm run build

The source code is under "/DataLineageApp/src/" folder. The webpack will build the code and copy all the required files for the application running to the folder "dist".

To debug locally, run the command:

npm start

To debug with watch-enabled, run the command:

npm run start:watch

this script will start webpack in watch mode and use nodemon to start server, so when there is any change on the client side, webpack will rebuild and copy the files, when there is a change on the server side, nodemon will find it, build and restart the server.

Deploy

  1. npm install in the root of the project
  2. npm run build
  3. copy the package.json to dist folder
  4. go to the dist folder and run npm install
  5. copy everything in dist folder and deploy them to the server.

The project is tested with Nodejs 8.11.2

Enjoy!

You can’t perform that action at this time.