AWS_ML_Crypto is a Spring-based JSON API written in Java that gets & stores historical cryptocurrency pricing and makes predictions predictions based on that data. This app utilizes a number of third-party tools, including CrytoCompare API, AWS SDK for Java, AWS Redshift & AWS Machine Learning Real Time Predictions. Using cron & the provided shell scripts, the user may continuously store new data, predict future values, and evaluate predictions.
Please note: This API is in active development.
The following must be installed on your local machine to use this program as is.
Clone the project to your desired location
$ git clone https://github.com/meejahnsnutshell/AWS_ML_Crypto
$ cd AWS_ML_Crypto
$ mvn clean install
You will need an AWS account. Please note that some services may incur fees. Avoid unnecessary charges by disabling services after use.
-
Set up your AWS Credentials.
-
Launch a Redshift Cluster. In the application.properties file of the application, update the datasource url, username, and password.
-
Follow these instructions to connect Workbench/J to your Redshift cluster. Build your tables.
-
Cryptocompare API is the source for historical data of current cryptocurrency pricing. All API calls for aggregating and loading data are within the HistoController. Calls that can be made include "data",
"backload", "backloadYear", and "coin". "coin" should be run first to populate data tables for coin list before "backloadYear" is used to backload the data for cryptocurrency wanted. -
Parameters for API call include type (type of historical call being made; default = histohour), fsym (from Symbol; default = BTC), tysm (to Symbol; default = USD), e (exchange; default = CCCAGG), extraParams (name of application), sign (If set to true, the server will sign the requests; default = false), tryConversion (If set to false, it will try to get values without using any conversion at all; default = true), aggregate (default = 1), limit (default = 50), toTs (unix Timestamp).
Required Parameters: Type, fsym, tsym, and e are the minimum parameters that should be included for all API calls.
- Ex of a URI call
.../histo/data?type=histohour&fsym=BTC&tsym=USD&limit=10&aggregate=1&e=CCCAGG
-
Data API calls mirror API calls from Cryptocompare. Data received will be checked against database for new data entries between timestamps. Any new data will be inserted into the database.
-
Backload API calls will check database for most current timestamp for requested cryptocurrency and will aggregate new data up to current time stamp and insert into database.
-
BackloadYear API call will aggregate all data since the requested year change based on current timestamp and cryptocurrency selected. Parameters are the same as other API calls with the addition of year (default value = -1). This parameter is the change to the current year date for data requested. Any new data that does not exist in the database will be entered. Ex. URI for backloading data for previous year:
/histo/backloadYear?type=histohour&fsym=BTC&tsym=USD&e=CCCAGG&limit=500&year=-1
- Coin API calls will get all current cryptocurrency listed on Cryptocompare and will insert any new currency into the database.
An AWS Machine Learning model is required to make a prediction. Models can be created in the AWS console (high customization)
qor programmatically (limited customization), but both require an existing datasource. Query parameters are createmodel,
modelId and modelName.
- To programmatically create a model, createmodel=true. Optional: modelId=xxxx&modelName=xxxx. Id and name are generated if not provided. Ex URIs:
.../predict/realtime?createmodel=true&modelId=1234&modelName=MyModel
or
.../predict/realtime?createmodel=true
Model parameters that can be configured in the code:
- To use an existing model, provide id and name.
.../predict/realtime?createmodel=false&modelid=1234&modelname=MyModel
- Results
Hourly highvalue prediction & evaluation
Shell scripts can be set with crontab -e to continuously get new data and make and analyze predictions.
- Ex hourly:
1 * * * * sh /path/to/bot.sh
2 * * * * sh /path/to/predict.sh <modelId> <modelName>
3 * * * * sh /path/to/analyze.sh
Note that timing is important. The scripts should be staggered and bot should not be run on the hour. Analyze must wait for the known value of the prediction to be received (analysis for a prediction made for 4PM can't be run until after 4PM.