Improve the balance output to ignore the coins that are in the Blacklist

[Th0masL]

Describe your environment

• Freqtrade Version: 2021.2

Describe the enhancement

Right now the command /balance is showing the balance for everything in the account.

But I believe that there's some improvement that we could do here.

Let's imagine that I use Binance for trading with freqtrade, but also to sometimes I buy and hold some specific coins. In this case, I will very likely want to be able to show the balance of all the things that are related to the freqtrade trading, but ignore the assets that I know are not related to trading.

For example, let's say that I trade using USDT as stake_currency, and let's say that I own some ETH and some BTC that plan to hold for long time, without trading them. Let's also assume that I have bought some BNB to be able to benefit lower trading fees, and in order to avoid to trade those coins, I have added the following values in the Pair Blacklist :

"ETH/USDT",
"BTC/USDT",
"BNB/USDT"
// or the wildcarded version, with XXX/.*

In this scenario, it would be very nice to have some sort of a /balance trading command that will ignore my ETH, BTC and BNB balances, and will only show me everything else, which is going to be everything that is related to the trading.

And I guess it would even make sense to "rename" the existing command to /balance all, and use the /balance command to default to only showing the balance of everything but the pairs that are in the Blacklist.

I think it shouldn't be too complicated to implement, so I can have a look at it if we believe this is something that would be useful.

Thoughts ?

[xmatthias]

i think it'll confuse more people than it will help

Especially the BNB balance you mention is quite important - as you should be aware when you run out of BNB to eventually buy more of these.

Usually, I'd also use the blacklist to forbid the bot to trade a coin i see as high risk, not one that i hold anyway.
Even if you hold ETH, and your USDT bot buys a bit more ETH, it'll only sell "the bot's" amount - not touching your balance (assuming you got BNB - otherwise it may use it to pay for the fees of this trade).

To correctly represent the "trading" balance, i'd therefore need to only report a subset of my available ETH balance (assume i personally own 1 ETH, and the bot bought 0.5 ETH temporarily), but not the full ETH balance - as this would then make the reported result quite inconsistent (it includes only what the bot uses - + my ETH balance).

This will make this a ... rather complex change i'd say, and one that's VERY prone to report false or missleading results.

[Th0masL]

Fair enough about keeping the BNB and the potential non-blacklisted currencies.

What about we add a * next to each blacklisted currency in the list, and a new line at the end, in the final summary, to also show the price without the blacklisted currencies ?

Like that :

BNB*:
 Available: 0.97716392
 Balance: 0.97716392
 Pending: 0.00000000
 Est. USDT: 226.513

USDT:
 Available: 214.57219155
 Balance: 214.57219155
 Pending: 0.00000000
 Est. USDT: 214.572

Estimated Value (incl. blacklisted):
 USDT:  441.085
 EUR: 371.477

Estimated Value (excl. blacklisted):
  USDT: 214.572
  EUR: 180.621

* blacklisted currency

[xmatthias]

my fear is that it'll make the output much more difficult to read / understand - without really providing much value.

you'll also end up with "problem" cases -

the bot can own a blacklisted pair (if it bought it before being blacklisted, for example) ... where would you list this? as blacklisted, or not (because the bot owns it) ?

these special cases will make it very difficult to both develop (that's probably okish - as it's a simple decision) - but even more difficult to understand from a user perspective.

[Th0masL]

I don't think it's making the output hard to read, but if it is, we could return this output only when using a special parameter, like /balances noblacklist or something like that, to allow users to use this output if they prefer it.

To he honest, my current /balance output is totally useless as I don't care about my BNB slowly reducing due to fees and the change of value due to the volatility of my BTC and ETH holdings are so much more impacting my total balance that I can barely see any amount related to freqtrade trading.

If I want to see my real, full, exchange wallet, I open the Binance app.

To me, the /balance command should only calculate the balance related to what freqtrade is used for, which is trading, and not HODLing.

[xmatthias]

To me, the /balance command should only calculate the balance related to what freqtrade is used for, which is trading, and not HODLing.

i don't necessarily mind changing how it works, but because of the above, you can't tie it to the blacklist.
Also, this is very specific to how you use the blacklist, and it will not give the same results than how i use the blacklist (for example) - as i allow the bot to trade pairs i HODL too.

If we assume the definition is to only show "related to freqtrade" - then it must handle at least the cases from above correctly (not sure what correct is yet in all cases):

• the account holds 1 ETH, the (this!) bot bought 0.1 ETH on top of this - the "only bot balances" one should NOT show 1.1, but only 0.1 ETH
• if the account owns RANDCOIN - which the bot does not trade (it's neither in white nor blacklist) - it should also not be shown (?)
• If it did buy 0.001 XRP and still owns it but the user deleted or manually closed the trade (unselable amount) - how would you handle this? Based on the above, you'd not show it - but that's incorrect as it's technically a balance the bot uses.

To me, this adds WAY too much complexity to a command that's supposed to give you a very quick overview - and is not relevant to the trading operations whatsoever.

[Th0masL]

I understand your points, and I agree that it will be hard to match all the corner cases.

I guess I'll simply write my own external python script that will calculate the balance the way I want it to be calculated, and send the message to Telegram once a day scheduled by a cronjob, that way I won't have to rely on using the /balance command.

On a side topic, how would you feel about allowing the user to add customized Telegram commands ? I.e. storing them in an external plugin or config file. Would you feel it is something that could be worth investigating ?

[xmatthias]

it's the corner cases -- and if you're a new user and look at your balance - but half of it is missing, you'll for sure be left wondering.

so far, i didn't think about telegram as extensible - but it might be an option - although i don't think many users will be able to - as telegram is quite ... picky in what it'll allow you to send unless it's properly / correctly escaped.

[Th0masL]

Actually I think I found the valid way to cover most -if not all- cases efficiently.

To get the real value of the balance dedicated to trading, I only has to add all the current values of the open trades (if any), and add the possible leftover of the stake_currency available on the exchange.

From an external script point of view that's a pretty straight forward calculation :

• use the rest_client.py and run status to get all the open trades, and take their current value (in stake currency)
• use the rest_client.py and run balances and only take the amount of the specific stake currency
• add them up, done, you get the value of your balance dedicated to trading at the date of now

My external script is scheduled to run every day at 00:01 AM and calculate the performance of the bot during the previous day, in percentage. I then save that into a JSON file so I can keep the historical values for each days, and I pretty print the result to Telegram as a nice table that shows the performance over the last 30 days.

Basically its kind of the same as the /daily command, except that I get the performance of each day as percentage, based on the value of the balance at the start of the day.

I found the /daily command nice but not really clear enough about the performance of the bot, since the fact that I have earned 30$ today doesn't mean that it was a good day or not, as it depends clearly on how much money I'm trading with.

[Th0masL]

I would love to have an SQL table that keep the open balance and close balance value for each day (in stake currency).

That would allow the user to get a whole new set of information about the bot's performance.

[xmatthias]

that's introducing another type of problem ...
if we say this runs at midnight ... which timezone? UTC? server time? user time? (we don't have that if it's different from server time).

also, around midnight is where most exchanges schedule downtimes ... so it will have some missing days from time to time (whenever there's a downtime or the bot wasn't running at that time) - both of which are points that need to be considered when consuming the data.

[Th0masL]

I'm currently executing it at midnight UTC, as I want all the data to be "timezone-neutral".

I've also made a security mechanism that keeps retrying to generate the performances of the previous day as long as it has not been generated already, meaning that if there's an exchange maintenance/outage from 23:00 UTC to 01:00 UTC, the performance calculation will be done somewhere after 01:00 UTC, when the exchange is back up and responding.

On my side, I'm totally fine with some small delay in the performance calculation in case of a maintenance/outage on the exchange.

[xmatthias]

well if you're interested in integrating this as base functionality (the /daily report is clearly lacking ..), i'd for sure not argue 😆

i think we'd want to store the full balances though, not just what the bot "owns" ... that might allow some "extended" reports as well in the future ... :)

[Th0masL]

well if you're interested in integrating this as base functionality (the /daily report is clearly lacking ..), i'd for sure not argue laughing

Yeah, I might take this task of improving the reports, but we first need to ensure that we agree on the requirements and on the expected result, as we need to find a way to get an output that satisfies both of us (me with my balance dedicated for trading, and you with your full balance).

i think we'd want to store the full balances though, not just what the bot "owns" ... that might allow some "extended" reports as well in the future ... :)

Do you mean that you want to store the full balances details ? Like each currency, with the amount the user owns, and also the estimated value ?

I don't really see what's the point of keeping the detail of what constitutes each day's amount, but if you have a precise and valid use case, I guess we could keep it.

I think we will want to keep both the "Trading Balance" and the "Total/Exchange Balance", so we can cover all the uses cases, therefore the content of the DB table could be something like below.

For the purpose of the examples below, let's imagine that on the Exchange there's $150 worth of a currency that we never trade, and to make the calculation easier, let's imagine that this currency price is not moving (i.e. BUSD). That means that there would always be +$150 on the Exchange Balances, compared to the Trading Balances.

Example of DB content :

Date        trade_open  trade_close  total_open  total_close
----------  ----------  -----------  ----------  -----------
2021-03-09        1200         1300        1350         1450
2021-03-08        1100         1200        1250         1350
2021-03-07        1250         1100        1400         1250
2021-03-06        1150         1250        1300         1400

When the user send the command /daily trading, then it would show the Daily Trading Balances, so the columns 1, 2 and 3 and the %Change column that would be calculated using 2 and 3.

When the user send the command /daily global, then it would show the Daily Global Balances, so the columns 1, 4 and 5 and the %Change column that would be calculated using 4 and 5.

Basically, the output that I'm expecting would be something like that :

---

Command : /daily balances trading

Output :
Daily Balances (Trading Only) over the last 7 days:

Date        Open USDT  Close USDT  %Change
----------  ---------  ----------  ------
2021-03-09       1200        1300    8.33
2021-03-08       1100        1200    9.09
2021-03-07       1250        1100  -12.00
2021-03-06       1150        1250    8.69
....

---

Command : /daily balances

Output :
Daily Balances over the last 7 days:

Date        Open USDT  Close USDT  %Change
----------  ---------  ----------  ------
2021-03-09       1350        1450    7.40
2021-03-08       1250        1350    8.00
2021-03-07       1400        1250  -10.71
2021-03-06       1300        1400    7.69
....

---

The %Change doesn't even have to be stored in the DB, as it can be calculated live when we are generating the Telegram message.

Also, we could decide to not show the Open column, and simply go with Date + Close USDT + %Change.

What do you think about that ?

[xmatthias]

personally, i'd say leave the "how to display" out for now - as there can be multiple different usecases for this, and calculating the "tradable" part might not be as easy as it sounds (we'll probably also need to include tradable_balance_ratio into this calculation - which i THINK you do with the 150$ in your example - but it'll not make this easier, not if you consider running 2 bots against the same account and same stake currency)

---

from the "balances" database, i'd not only store the stake-currency - but all balances (eventually we might end up having 2 tables, one with "all" balances, and one as you describe above - which shows it in stake_currency based on the current (that day's) conversion rate.

While the "full" table might not be useful for the telegram output itself (not in the form you describe it above) - we could run /balance 2021-01-01 to get the balance on that day.

apart from the potential "additional" table (i see the usage for that mainly with the rest api to potentially generate graphs of the owned currencies, not with telegram), i think your approach should work.

The %change is definitely not necessary in the table, that is too easy to calculate when gathering the output.

You might want to be careful with the datatype of the "date" column - i would use "DateTime" not "Date" - as it will allow us to increase the granularity in the future.

[Th0masL]

Do we agree that if, for each day (let's say at 00:01 AM UTC), we store :

• the equivalent of full output of the current /balances command
• the equivalent of full output of the current /status command
• the name of the current stake_currency
• the value of tradable_balance_ratio

Then we will be able to calculate absolutely everything we will ever need related to balances and daily reports ?

I think it would even allow to generate data that is able to support the cases where an user is running multiple bots with the same stake_currency.

It might be a lot of data to store every day, but at the same time it would indeed give us the freedom to generate all kind of historical reports and calculations.

Are you OK in storing such amount of data every day ? (I think we don't have much choice, tbh)

[xmatthias]

the equivalent of full output of the current /status command

what would we need this data for?
this should be covered by what we have in the trades table ...

[Th0masL]

Yes, of course we won't need exactly everything, but basically we will need to keep quite many information from /status, unless you want to query the existing data from the trades table ?

The problem with that is that we need to keep a "snapshot" of the values for each trades at the time of the new day (let's say 00:01 AM UTC), and I think we don't have this information at this precise time of 00:01 AM in the trades table, right ?

Maybe you want us to add more data to the trades table exactly at the time of 00:01 AM, and then use this data ?

I'm going to take some time and try to prepare an example of what could be the table structure, with some data examples, and I will post that here to confirm.

[xmatthias]

you can't use the trades table for sure - assuming a trade goes over multiple days - you can't have unlimited columns to cover for all days.

i'm still not sure which info you'd need for this ... i think having the balance at that time will be enough (you can still cross-reference that with the trades table to determine if it's one of your trades or if it isn't).