Skip to content

Commit

Permalink
Dust protection (#214)
Browse files Browse the repository at this point in the history
* fix undefined gasPrice

* load file with threshold data

* add table to queries

* add minSentinelBalanceThreshold to file

* get minSentinelBalanceThreshold as config var

* create balance alert every 5min

* only one logic liquidation, for both cases.

* adds endpoint liquidationscount that will return workload of 7days

* nextliquidations endpoint with configurable timeframe, replaces dedicated endpoint for counting (will add scripts wrapping jq queries achieving the same)

* added script for querying the nextliquidations endpoint with jq

* use the script without npm in order to avoid noise in the output

* fix

---------

Co-authored-by: Didi <didi@superfluid.finance>
Co-authored-by: didi <git@d10r.net>
  • Loading branch information
3 people committed Jun 6, 2023
1 parent 620de80 commit 94a9e83
Show file tree
Hide file tree
Showing 9 changed files with 279 additions and 16 deletions.
8 changes: 7 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,13 @@ systemctl enable superfluid-sentinel.service
### Monitoring & Alerting

The sentinel can provide monitoring information. In the default configuration, this is available on port 9100 and json formatted.
This includes a flag "healthy" which turns to `false` in case of malfunction, e.g. if there's a problem with the local DB or with the RPC connection.

The endpoint `/` returns a status summary, including a flag `healthy` which turns to `false` in case of malfunction, e.g. if there's a problem with the local DB or with the RPC connection.
The endpoint `/nextliquidations` returns a list accounts likely up for liquidation next. The timeframe for that preview defaults to 1h. You can set a custom timeframe by setting an url parameter `timeframe`. Supported units are m (minutes), h (hours), d(days), w(weeks), M(months), y(years). Example: `/nextliquidations?timeframe=3d`

Using the json parsing tool `jq`, you can pretty-print the output of the metris endpoint and also run queries on it.
There's also a convenience script available with a few potentially useful queries, see `scripts/query-metrics.sh --help`.
(Note: this script doesn't use the ENV vars related to metrics from the .env file - you may need to set the HOST env var).

You can also set up notifications to Slack or Telegram. Events triggering a notification include
* sentinel restarts
Expand Down
148 changes: 148 additions & 0 deletions scripts/query-metrics.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,148 @@
#!/bin/bash

# requires curl and jq

set -e
set -o pipefail

host=${HOST:-http://localhost:9100}
timeframe=${TIMEFRAME:-"1h"}

# Check if curl is installed
if ! command -v curl &> /dev/null
then
echo "curl could not be found"
exit 1
fi

# Check if jq is installed
if ! command -v jq &> /dev/null
then
echo "jq could not be found"
exit 1
fi

# Function to display help
function display_help() {
echo "Usage: $0 <command> [argument(s)]"
echo
echo "Commands:"
echo " streams List of streams due for liqudation in the given timeframe"
echo " nr-streams Number of streams due for liqudation in the given timeframe"
echo " streams-filter-token <token> List of streams due, filtered by SuperToken address"
echo " streams-filter-sender <sender> List of streams due, filtered by sender address"
echo " streams-filter-receiver <receiver> List of streams due, filtered by receiver address"
echo " streams-above-flowrate <flowrate> List of streams due, filtered by flowrate being above (wad/seconds)"
echo " streams-above-flowrate <flowrate> List of streams due, filtered by flowrate being below (wad/seconds)"
echo " nr-streams-by-token Number of streams due, grouped by SuperToken"
echo " nr-streams-by-sender Number of streams due, grouped by sender"
echo " nr-streams-by-receiver Number of streams due, grouped by receiver"
echo
echo "ENV vars:"
echo " HOST: use a host other than 'http://localhost:9100'"
echo " TIMEFRAME: use a timeframe other than '1h' - supported units range from minutes to years: m, h, d, w, M, y"
echo " DEBUG: if set, the command executed is printed before its execution. This can be useful if you want to modify a query"
echo
echo "Note that you can further process all list formatted output with jq, e.g. in order to count items returned by a filter query just append ' | jq length'."
echo "This works only if the DEBUG var is not set"
exit 0
}

# Check if no command is provided
if [ $# -eq 0 ]; then
echo "No command provided"
display_help
exit 1
fi

# Process command
case "$1" in
streams)
cmd="curl -s $host/nextliquidations?timeframe=$timeframe | jq"
[[ $DEBUG ]] && echo "$cmd"
eval "$cmd" || echo "### FAILED"
;;
nr-streams)
cmd="curl -s $host/nextliquidations?timeframe=$timeframe | jq length"
[[ $DEBUG ]] && echo "$cmd"
eval "$cmd" || echo "### FAILED"
;;
streams-filter-token)
if [ $# -eq 2 ]; then
token=${2,,} # convert to lowercase
cmd="curl -s $host/nextliquidations?timeframe=$timeframe | jq '.[] | select(.superToken | ascii_downcase == \"$token\")'"
[[ $DEBUG ]] && echo "$cmd"
eval "$cmd" || echo "### FAILED"
else
echo "Error: '$1' requires a token (address) argument"
exit 1
fi
;;
streams-filter-sender)
if [ $# -eq 2 ]; then
sender=${2,,} # convert to lowercase
cmd="curl -s $host/nextliquidations?timeframe=$timeframe | jq '.[] | select(.sender | ascii_downcase == \"$sender\")'"
[[ $DEBUG ]] && echo "$cmd"
eval "$cmd" || echo "### FAILED"
else
echo "Error: '$1' requires a sender (address) argument"
exit 1
fi
;;
streams-filter-receiver)
if [ $# -eq 2 ]; then
receiver=${2,,} # convert to lowercase
cmd="curl -s $host/nextliquidations?timeframe=$timeframe | jq '.[] | select(.receiver | ascii_downcase == \"$receiver\")'"
[[ $DEBUG ]] && echo "$cmd"
eval "$cmd" || echo "### FAILED"
else
echo "Error: '$1' requires a receiver (address) argument"
exit 1
fi
;;
streams-above-flowrate)
if [ $# -eq 2 ]; then
flowrate=$2
cmd="curl -s $host/nextliquidations?timeframe=$timeframe | jq '[.[] | select(.flowRate > $flowrate)]'"
[[ $DEBUG ]] && echo "$cmd"
eval "$cmd" || echo "### FAILED"
else
echo "Error: '$1' requires a flowrate (wad/second) argument"
exit 1
fi
;;
streams-below-flowrate)
if [ $# -eq 2 ]; then
flowrate=$2
cmd="curl -s $host/nextliquidations?timeframe=$timeframe | jq '[.[] | select(.flowRate < $flowrate)]'"
[[ $DEBUG ]] && echo "$cmd"
eval "$cmd" || echo "### FAILED"
else
echo "Error: '$1' requires a flowrate (wad/second) argument"
exit 1
fi
;;
nr-streams-by-token)
cmd="curl -s $host/nextliquidations?timeframe=$timeframe | jq 'group_by(.superToken) | map({superToken: .[0].superToken, count: length})'"
[[ $DEBUG ]] && echo "$cmd"
eval "$cmd" || echo "### FAILED"
;;
nr-streams-by-sender)
cmd="curl -s $host/nextliquidations?timeframe=$timeframe | jq 'group_by(.sender) | map({sender: .[0].sender, count: length})'"
[[ $DEBUG ]] && echo "$cmd"
eval "$cmd" || echo "### FAILED"
;;
nr-streams-by-receiver)
cmd="curl -s $host/nextliquidations?timeframe=$timeframe | jq 'group_by(.receiver) | map({receiver: .[0].receiver, count: length})'"
[[ $DEBUG ]] && echo "$cmd"
eval "$cmd" || echo "### FAILED"
;;
--help)
display_help
;;
*)
echo "Unknown command: $1"
display_help
exit 1
;;
esac
21 changes: 18 additions & 3 deletions src/app.js
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ const HTTPServer = require("./httpserver/server");
const Report = require("./httpserver/report");
const Notifier = require("./services/notifier");
const SlackNotifier = require("./services/slackNotifier");
const TelegramNotfier = require("./services/telegramNotifier");
const TelegramNotifier = require("./services/telegramNotifier");
const NotifierJobs = require("./services/notificationJobs");
const Errors = require("./utils/errors/errors");
const { wad4human } = require("@decentral.ee/web3-helpers");
Expand All @@ -39,7 +39,8 @@ class App {
FlowUpdatedModel: require("./database/models/flowUpdatedModel")(this.db),
SuperTokenModel: require("./database/models/superTokenModel")(this.db),
SystemModel: require("./database/models/systemModel")(this.db),
UserConfig: require("./database/models/userConfiguration")(this.db)
UserConfig: require("./database/models/userConfiguration")(this.db),
ThresholdModel: require("./database/models/thresholdModel")(this.db),
}
this.db.queries = new Repository(this);
this.eventTracker = new EventTracker(this);
Expand Down Expand Up @@ -68,9 +69,10 @@ class App {
this._slackNotifier = new SlackNotifier(this, {timeout: 3000});
}
if (this.config.TELEGRAM_BOT_TOKEN && this.config.TELEGRAM_CHAT_ID) {
this._telegramNotifier = new TelegramNotfier(this);
this._telegramNotifier = new TelegramNotifier(this);
}
if (this._slackNotifier || this._telegramNotifier) {
this.logger.info("initializing notification jobs")
this.notificationJobs = new NotifierJobs(this);
}

Expand Down Expand Up @@ -206,6 +208,19 @@ class App {
process.exit(1);
}
await this.db.queries.saveConfiguration(JSON.stringify(userConfig));
// get json file with tokens and their thresholds limits. Check if it exists and loaded to json object
try {
const thresholds = require("../thresholds.json");
const tokensThresholds = thresholds.networks[await this.client.getChainId()];
this.config.SENTINEL_BALANCE_THRESHOLD = tokensThresholds.minSentinelBalanceThreshold;
// update thresholds on database
await this.db.queries.updateThresholds(tokensThresholds.thresholds);
} catch (err) {
this.logger.warn(`error loading thresholds.json`);
await this.db.queries.updateThresholds({});
}


// collect events to detect superTokens and accounts
const currentBlock = await this.loadEvents.start();
// query balances to make liquidations estimations
Expand Down
12 changes: 12 additions & 0 deletions src/database/models/thresholdModel.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
const Sequelize = require("sequelize");
module.exports = (db) => { return db.define("thresholds", {
address: {
type: Sequelize.STRING,
allowNull: false,
primaryKey: true
},
above:{
type: Sequelize.INTEGER,
defaultValue: 0
},
})};
45 changes: 37 additions & 8 deletions src/database/repository.js
Original file line number Diff line number Diff line change
Expand Up @@ -68,9 +68,11 @@ class Repository {
});
}

async getLiquidations(checkDate, onlyTokens, excludeTokens, limitRows) {
// liquidations where flowRate is above a certain threshold
async getLiquidations(checkDate, onlyTokens, excludeTokens, limitRows, useThresholds = true) {
let inSnipped = "";
let inSnippedLimit = "";

// if configured onlyTokens we don't filter by excludeTokens
const tokenFilter = onlyTokens !== undefined ? onlyTokens : excludeTokens;
if (tokenFilter !== undefined) {
Expand All @@ -80,18 +82,24 @@ class Repository {
inSnippedLimit = `LIMIT ${limitRows}`;
}

const joinThresholds = useThresholds ? 'LEFT JOIN thresholds thr on agr.superToken = thr.address' : '';
const flowRateCondition = useThresholds ? 'out.flowRate >= COALESCE(out.above, 0)' : 'out.flowRate > 0';

const sqlquery = `SELECT * FROM (SELECT agr.superToken, agr.sender, agr.receiver,
CASE pppmode
WHEN 0 THEN est.estimation
WHEN 1 THEN est.estimationPleb
WHEN 2 THEN est.estimationPirate
END as estimation,
pppmode
pppmode,
flowRate,
${useThresholds ? 'COALESCE(thr.above, 0) as above' : '0 as above'}
FROM agreements agr
INNER JOIN supertokens st on agr.superToken == st.address
INNER JOIN estimations est ON agr.sender = est.address AND agr.superToken = est.superToken AND est.estimation <> 0 AND agr.flowRate <> 0
INNER JOIN estimations est ON agr.sender = est.address AND agr.superToken = est.superToken AND est.estimation <> 0
${joinThresholds}
) AS out
WHERE out.estimation <= :dt ${inSnipped}
WHERE ${flowRateCondition} AND out.estimation <= :dt ${inSnipped}
ORDER BY out.estimation ASC ${inSnippedLimit}`;

if (inSnipped !== "") {
Expand All @@ -110,25 +118,32 @@ ORDER BY out.estimation ASC ${inSnippedLimit}`;
});
}

async getNumberOfBatchCalls(checkDate, onlyTokens, excludeTokens) {
async getNumberOfBatchCalls(checkDate, onlyTokens, excludeTokens, useThresholds = true) {
let inSnipped = "";
// if configured onlyTokens we don't filter by excludeTokens
const tokenFilter = onlyTokens !== undefined ? onlyTokens : excludeTokens;
if (tokenFilter !== undefined) {
inSnipped = `and out.superToken ${ onlyTokens !== undefined ? "in" : "not in" } (:tokens)`;
}

const joinThresholds = useThresholds ? 'LEFT JOIN thresholds thr on agr.superToken = thr.address' : '';
const flowRateCondition = useThresholds ? 'out.flowRate >= COALESCE(out.above, 0)' : 'out.flowRate > 0';

const sqlquery = `SELECT superToken, count(*) as numberTxs FROM (SELECT agr.superToken, agr.sender, agr.receiver,
CASE pppmode
WHEN 0 THEN est.estimation
WHEN 1 THEN est.estimationPleb
WHEN 2 THEN est.estimationPirate
END as estimation
END as estimation,
pppmode,
flowRate,
${useThresholds ? 'COALESCE(thr.above, 0) as above' : '0 as above'}
FROM agreements agr
INNER JOIN supertokens st on agr.superToken == st.address
INNER JOIN estimations est ON agr.sender = est.address AND agr.superToken = est.superToken AND est.estimation <> 0 AND agr.flowRate <> 0
INNER JOIN estimations est ON agr.sender = est.address AND agr.superToken = est.superToken AND est.estimation <> 0
${joinThresholds}
) AS out
WHERE out.estimation <= :dt ${inSnipped}
WHERE ${flowRateCondition} AND out.estimation <= :dt ${inSnipped}
group by out.superToken
having count(*) > 1
order by count(*) desc`;
Expand Down Expand Up @@ -196,6 +211,20 @@ order by count(*) desc`;
type: QueryTypes.SELECT
});
}

async updateThresholds(thresholds) {
await this.app.db.models.ThresholdModel.destroy({truncate: true});
// check if thresholds is empty object
if(Object.keys(thresholds).length === 0) {
// create table without table data
return this.app.db.models.ThresholdModel.sync();
} else {
// from json data save it to table
for (const threshold of thresholds) {
await this.app.db.models.ThresholdModel.create(threshold);
}
}
}
}

module.exports = Repository;
35 changes: 34 additions & 1 deletion src/httpserver/server.js
Original file line number Diff line number Diff line change
Expand Up @@ -28,12 +28,45 @@ class HTTPServer {
}
});

// helper function for argument parsing
const parseTimeframe = (timeframe) => {
const units = {
'm': 60,
'h': 3600,
'd': 3600 * 24,
'w': 3600 * 24 * 7,
'M': 3600 * 24 * 30,
'y': 3600 * 24 * 365,
};

const regex = /^(\d+)([mhdwMy])$/;
const match = timeframe.match(regex);

if (match) {
const value = parseInt(match[1], 10);
const unit = match[2];
return value * units[unit];
}

return null;
};

// get a list of upcoming liquidations - configurable timeframe, defaults to 1h
this.server.get("/nextliquidations", async (req, res) => {
const timeframeParam = req.query.timeframe || '1h';
const timeframeInSeconds = parseTimeframe(timeframeParam);

if (timeframeInSeconds === null) {
res.status(400).send({ message: 'Invalid timeframe format. Use a value like "2h", "5d", etc.' });
return;
}

const liquidations = await this.app.db.queries.getLiquidations(
this.app.time.getTimeWithDelay(-3600),
this.app.time.getTimeWithDelay(-timeframeInSeconds),
this.app.config.TOKENS,
this.app.config.EXCLUDED_TOKENS
);

try {
res.send(liquidations);
} catch (e) {
Expand Down
11 changes: 8 additions & 3 deletions src/services/notificationJobs.js
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
const BN = require("bn.js");
const {wad4human} = require("@decentral.ee/web3-helpers/src/math-utils");

const timeout = ms => new Promise(resolve => setTimeout(resolve, ms));
async function trigger (obj, ms) {
await timeout(ms);
Expand All @@ -17,11 +17,16 @@ class NotificationJobs {
const healthData = `Healthy: ${healthcheck.healthy}\nChainId: ${healthcheck.network.chainId}`;
this.app.notifier.sendNotification(healthData);
}
const accountBalance = await this.app.client.getAccountBalance();
if(new BN(accountBalance).lt(new BN(this.app.config.SENTINEL_BALANCE_THRESHOLD))) {
const balanceData = `Attention: Sentinel balance: ${wad4human(accountBalance)}`;
this.app.notifier.sendNotification(balanceData);
}
}

async start () {
// run every hour ( value in ms)
this.run(this, 3600*1000);
// run every 5 min ( value in ms)
this.run(this, 300000);
}

async run (self, time) {
Expand Down
Loading

0 comments on commit 94a9e83

Please sign in to comment.