Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
c982072
commit c027d97
Showing
27 changed files
with
580 additions
and
584 deletions.
There are no files selected for viewing
16 changes: 0 additions & 16 deletions
16
.github/ISSUE_TEMPLATE/algorithm-development-and-backtesting.md
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,98 +1,99 @@ | ||
| # README | ||
|
|
||
| [](https://diadata.readthedocs.io/en/latest/?badge=latest) | ||
|
|
||
| # DIA: Technical Introduction | ||
| Welcome to the [DIAdata.org](https://diadata.org/) platform. | ||
| DIA is an ecosystem for open financial data in a financial smart contract ecosystem. | ||
| The target of DIA is to bring together data analysts, data providers and data users. | ||
| In general, DIA provides a reliable and verifiable bridge between off-chain data from various sources and on-chain smart contracts that can be used to build a variety of financial dApps. | ||
|  | ||
| ## DIA: Technical Introduction | ||
|
|
||
| Welcome to the [DIAdata.org](https://diadata.org/) platform. DIA is an ecosystem for open financial data in a financial smart contract ecosystem. The target of DIA is to bring together data analysts, data providers and data users. In general, DIA provides a reliable and verifiable bridge between off-chain data from various sources and on-chain smart contracts that can be used to build a variety of financial dApps.  | ||
|
|
||
| #### Decentralized Financial Data | ||
|
|
||
| ### Decentralized Financial Data | ||
| All DIA data points can be used in Ethereum Smart Contracts. | ||
| We provide oracle solutions to use this financial information in any smart contract. | ||
| The correctness of oracle data can always be verified by monitoring our off-chain real-time API and the data published in the oracle. | ||
| All DIA data points can be used in Ethereum Smart Contracts. We provide oracle solutions to use this financial information in any smart contract. The correctness of oracle data can always be verified by monitoring our off-chain real-time API and the data published in the oracle. | ||
|
|
||
| ### Alpha Version - dApp Registry | ||
|
|
||
| ## Alpha Version - dApp Registry | ||
| * [CoinHub](http://coinhub.diadata.org) - A transparent, open-source alternative to CMC | ||
| * [SIX/F10 PSD II Sandbox](https://f10-sandbox-portal.apps.ndgit.com/#/apis) - PSD II and pricing data for FinTechs | ||
| * [Neufund](https://blog.neufund.org/neufund-1-1-is-live-on-mainnet-db35d77b2ef2) | ||
| * [Helena/Consensys](https://helena.network) - Delivering real-time blockchain fundamental insights | ||
| * [Open-Source Calculation Agent](https://blockstate.com/decentral-calculation-agent/) | ||
| * [Swiss ISIN note using DIA data](https://en.bitcoinwiki.org/wiki/Index_Fund) | ||
|
|
||
| ## Methodology Documentation | ||
| Currently, DIA collects several classes of data. | ||
| Separate documentation exists for each of these different use cases. | ||
| ### Methodology Documentation | ||
|
|
||
| Currently, DIA collects several classes of data. Separate documentation exists for each of these different use cases. | ||
|
|
||
| * [Exchange Scrapers](documentation/methodology/ExchangePrices.md) are the components that get trade data from various exchanges and use these trades to construct average prices. | ||
| * [Supply Scrapers](documentation/methodology/SupplyNumbers.md) are used to determine circulating supply numbers for cryptocurrency assets. | ||
| * [Oracles](documentation/methodology/Oracles.md) show how our oracles can be accessed. | ||
| This information varies for each currency and is highly specific. | ||
| * [Exchange Scrapers](documentation/methodology/exchangeprices.md) are the components that get trade data from various exchanges and use these trades to construct average prices. | ||
| * [Supply Scrapers](documentation/methodology/supplynumbers.md) are used to determine circulating supply numbers for cryptocurrency assets. | ||
| * [Oracles](documentation/methodology/oracles.md) show how our oracles can be accessed. | ||
|
|
||
| ## Coinhub Oracles | ||
| This information varies for each currency and is highly specific. | ||
|
|
||
| The Coinhub oracles are located in a [single smart contract](https://etherscan.io/address/0xD47FDf51D61c100C447E2D4747c7126F19fa23Ef). | ||
| This smart contract holds mappings from an index to an asset's name, its price, and supply data. | ||
| By using `getParameters(asset_index)` it is possible to retrieve this data. | ||
| Along with the actual data there is a metadata field for the timestamp of the last update. | ||
| ### Coinhub Oracles | ||
|
|
||
| Updates are supplied by the DIA Oracle service that periodically supplies updates into the smart contract. | ||
| In the event view, the latest updates can be seen. | ||
| The event fields show the values in the following locations: | ||
| The Coinhub oracles are located in a [single smart contract](https://etherscan.io/address/0xD47FDf51D61c100C447E2D4747c7126F19fa23Ef). This smart contract holds mappings from an index to an asset's name, its price, and supply data. By using `getParameters(asset_index)` it is possible to retrieve this data. Along with the actual data there is a metadata field for the timestamp of the last update. | ||
|
|
||
| * Price in Cent is in the second field (display as `number`) | ||
| * Supply is in the third field (display as `number`) | ||
| * UNIX Timestamp of last update is displayed in the fourth field (display as `number`) | ||
| * Name is displayed in the sixth and last field (display as `text`). | ||
| Updates are supplied by the DIA Oracle service that periodically supplies updates into the smart contract. In the event view, the latest updates can be seen. The event fields show the values in the following locations: | ||
|
|
||
| ## API Access | ||
| You can find documentation for our [live API on our api documentation site](api/docs/api.md). | ||
| Several endpoints exist that show the different kind of data we collect: | ||
| * Price in Cent is in the second field \(display as `number`\) | ||
| * Supply is in the third field \(display as `number`\) | ||
| * UNIX Timestamp of last update is displayed in the fourth field \(display as `number`\) | ||
| * Name is displayed in the sixth and last field \(display as `text`\). | ||
|
|
||
| When referencing currencies, `{TLA}` has to be replaced by the acronym for an asset (e.g., BTC for Bitcoin). | ||
| * Quotations for prices (SMA120) can be found at endpoint [https://api.diadata.org/v1/symbol/BTC](https://api.diadata.org/v1/symbol/BTC). | ||
| ### API Access | ||
|
|
||
| You can find documentation for our [live API on our api documentation site](api/docs/api.md). Several endpoints exist that show the different kind of data we collect: | ||
|
|
||
| When referencing currencies, `{TLA}` has to be replaced by the acronym for an asset \(e.g., BTC for Bitcoin\). | ||
|
|
||
| * Quotations for prices \(SMA120\) can be found at endpoint [https://api.diadata.org/v1/symbol/BTC](https://api.diadata.org/v1/symbol/BTC). | ||
| * Supply data for crypto assets is located at [https://api.diadata.org/v1/supply/BTC](https://api.diadata.org/v1/supply/BTC) | ||
|
|
||
| Currently, DIA collects data from several crypto exchanges. | ||
| To get an overview, the latest information about these exchanges can be found in this table: | ||
|
|
||
| Name | Number of Pairs | API link | ||
| --- | --- | --- | ||
| Binance | [47](config/Binance.json) | [Binance API Documentation](https://github.com/binance-exchange/binance-official-api-docs) | ||
| Bitfinex | [25](config/Bitfinex.json) | [Bitfinex Websocket Documentation](https://docs.bitfinex.com/docs/ws-general) | ||
| Coinbase | [3](config/CoinBase.json) | [Coinbase v2 API](https://developers.coinbase.com/api/v2) | ||
| GateIO | [42](config/GateIO.json) | [Gate.io API Documentation](https://www.gate.io/api2) | ||
| HitBTC | [43](config/HitBTC.json) | [HitBTC API Browser](https://api.hitbtc.com/api/2/explore/) | ||
| Huobi | [40](config/Huobi.json) | [Huobi API Documentation](https://github.com/huobiapi/API_Docs_en/wiki/Huobi.pro-API) | ||
| Kraken | [8](config/Kraken.json) | [Kraken Public API Documentation](https://www.kraken.com/help/api#public-market-data) | ||
| LBank | [20](config/LBank.json) | [LBank API Documentation (Chinese)](https://github.com/LBank-exchange/lbank-official-api-docs) | ||
| OKEx | [27](config/OKEx.json) | [OKEx API Documentation (Chinese)](https://github.com/okcoin-okex/API-docs-OKEx.com) | ||
| Simex | [14](config/Simex.json) | [Simex API Documentation](https://simex.global/en/docs/introduction) | ||
| ZB.com | [127](config/ZB.json) | [Zb.com API Documentation (Chinese)](https://www.zb.com/i/developer) | ||
| Currently, DIA collects data from several crypto exchanges. To get an overview, the latest information about these exchanges can be found in this table: | ||
|
|
||
| | Name | Number of Pairs | API link | | ||
| | :--- | :--- | :--- | | ||
| | Binance | [47](https://github.com/diadata-org/diadata/tree/c982072de2ac488c5f0bdf32b677cbac1965583e/config/Binance.json) | [Binance API Documentation](https://github.com/binance-exchange/binance-official-api-docs) | | ||
| | Bitfinex | [25](https://github.com/diadata-org/diadata/tree/c982072de2ac488c5f0bdf32b677cbac1965583e/config/Bitfinex.json) | [Bitfinex Websocket Documentation](https://docs.bitfinex.com/docs/ws-general) | | ||
| | Coinbase | [3](https://github.com/diadata-org/diadata/tree/c982072de2ac488c5f0bdf32b677cbac1965583e/config/CoinBase.json) | [Coinbase v2 API](https://developers.coinbase.com/api/v2) | | ||
| | GateIO | [42](https://github.com/diadata-org/diadata/tree/c982072de2ac488c5f0bdf32b677cbac1965583e/config/GateIO.json) | [Gate.io API Documentation](https://www.gate.io/api2) | | ||
| | HitBTC | [43](https://github.com/diadata-org/diadata/tree/c982072de2ac488c5f0bdf32b677cbac1965583e/config/HitBTC.json) | [HitBTC API Browser](https://api.hitbtc.com/api/2/explore/) | | ||
| | Huobi | [40](https://github.com/diadata-org/diadata/tree/c982072de2ac488c5f0bdf32b677cbac1965583e/config/Huobi.json) | [Huobi API Documentation](https://github.com/huobiapi/API_Docs_en/wiki/Huobi.pro-API) | | ||
| | Kraken | [8](https://github.com/diadata-org/diadata/tree/c982072de2ac488c5f0bdf32b677cbac1965583e/config/Kraken.json) | [Kraken Public API Documentation](https://www.kraken.com/help/api#public-market-data) | | ||
| | LBank | [20](https://github.com/diadata-org/diadata/tree/c982072de2ac488c5f0bdf32b677cbac1965583e/config/LBank.json) | [LBank API Documentation \(Chinese\)](https://github.com/LBank-exchange/lbank-official-api-docs) | | ||
| | OKEx | [27](https://github.com/diadata-org/diadata/tree/c982072de2ac488c5f0bdf32b677cbac1965583e/config/OKEx.json) | [OKEx API Documentation \(Chinese\)](https://github.com/okcoin-okex/API-docs-OKEx.com) | | ||
| | Simex | [14](https://github.com/diadata-org/diadata/tree/c982072de2ac488c5f0bdf32b677cbac1965583e/config/Simex.json) | [Simex API Documentation](https://simex.global/en/docs/introduction) | | ||
| | ZB.com | [127](https://github.com/diadata-org/diadata/tree/c982072de2ac488c5f0bdf32b677cbac1965583e/config/ZB.json) | [Zb.com API Documentation \(Chinese\)](https://www.zb.com/i/developer) | | ||
|
|
||
| Here you can find the introduction to the structure of the system and first steps to contribute. | ||
|
|
||
| ## DIA Technical Structure | ||
| DIA is setup as a hybrid system with off-chain components for storing and proceccing large amounts of data and on-chain components providing data sources for financial smart contracts. | ||
| Currently, Ethereum is used as smart contract plaform because of its widespread use and technical quality, but the DIA system is not limited to one blockchain technology. | ||
| ### DIA Technical Structure | ||
|
|
||
| DIA is setup as a hybrid system with off-chain components for storing and proceccing large amounts of data and on-chain components providing data sources for financial smart contracts. Currently, Ethereum is used as smart contract plaform because of its widespread use and technical quality, but the DIA system is not limited to one blockchain technology. | ||
|
|
||
| ### Centralized Backend | ||
| For collecting financial data, we use a centralized backend that runs collectors for all kinds of financial data. | ||
| All collected data is processed by a database setup consisting of a stream-oriented kafka instance and a key-value store for faster access of certain intermediate results. | ||
| These collectors are separated in different classes: | ||
| #### Centralized Backend | ||
|
|
||
| For collecting financial data, we use a centralized backend that runs collectors for all kinds of financial data. All collected data is processed by a database setup consisting of a stream-oriented kafka instance and a key-value store for faster access of certain intermediate results. These collectors are separated in different classes: | ||
|
|
||
| * Exchange Scrapers: These collect data from exchanges for cryptocurrencies and other assets \(like stocks, futures, and rare earth metals\). | ||
|
|
||
| Every exchange scraper is run independently and is structured around the idea of `pairs` that are used to indicate the pair of assets that is used in a trade. | ||
|
|
||
| For example, a trade from Bitcoin to US-Dollar is stored under `BTCUSD`. | ||
|
|
||
| * Exchange Scrapers: These collect data from exchanges for cryptocurrencies and other assets (like stocks, futures, and rare earth metals). | ||
| Every exchange scraper is run independently and is structured around the idea of `pairs` that are used to indicate the pair of assets that is used in a trade. | ||
| For example, a trade from Bitcoin to US-Dollar is stored under `BTCUSD`. | ||
| * Blockchain Scrapers: These scrapers are used to determine attributes of blockchains. The most prominent attribute is the circulating supply of a cryptocurrency, for which various scrapers already exist. | ||
| Depending on the expected update rate of a blockchain, these scrapers update the backend periodically about their data. | ||
| Blockchain scrapers are run as docker containers that can be managed independently. | ||
| Often, these collect Gigabytes of data over time (they are dealing with ever-growing blockchains) and thus should be managed in a separate way not directly baked in the database executable. | ||
|
|
||
| Depending on the expected update rate of a blockchain, these scrapers update the backend periodically about their data. | ||
|
|
||
| Blockchain scrapers are run as docker containers that can be managed independently. | ||
|
|
||
| Often, these collect Gigabytes of data over time \(they are dealing with ever-growing blockchains\) and thus should be managed in a separate way not directly baked in the database executable. | ||
|
|
||
| * Quotation Scrapers: They are used to collect official quotations from central trusted providers. | ||
| The first quotations in our database are daily exchange rates from the European Central Bank (ECB) against various international currencies. | ||
|
|
||
| ## How to contribute | ||
| DIA is open for anyone to contribute and we always welcome pull requests. | ||
| We also provide tasks on [Gitcoin](https://gitcoin.co/), a collaboration tool focused on cryptocurrency projects. | ||
| The first quotations in our database are daily exchange rates from the European Central Bank \(ECB\) against various international currencies. | ||
|
|
||
| ### How to contribute | ||
|
|
||
| DIA is open for anyone to contribute and we always welcome pull requests. We also provide tasks on [Gitcoin](https://gitcoin.co/), a collaboration tool focused on cryptocurrency projects. | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| # Table of contents | ||
|
|
||
| * [README](README.md) | ||
| * [documentation](documentation/README.md) | ||
| * [DIAdata.org API](documentation/api.md) | ||
| * [DIA data documentation](documentation/index.md) | ||
| * [tutorials](documentation/tutorials/README.md) | ||
| * [Write your own blockchain scraper](documentation/tutorials/blockchainscraper.md) | ||
| * [Write your own exchange scraper](documentation/tutorials/exchangescrapers.md) | ||
| * [methodology](documentation/methodology/README.md) | ||
| * [Circulating Supply Numbers](documentation/methodology/supplynumbers.md) | ||
| * [Oracles in DIA](documentation/methodology/oracles.md) | ||
| * [Pricing Methodology of Crypto Exchange Quotations](documentation/methodology/exchangeprices.md) | ||
| * [api](api/README.md) | ||
| * [docs](api/docs/README.md) | ||
| * [diadata.org API](api/docs/api.md) | ||
| * [Contributor Covenant Code of Conduct](code_of_conduct.md) | ||
| * [Contributing](contributing.md) | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| # api | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| # docs | ||
|
|
Oops, something went wrong.