Add fee suggestions to /fee_stats endpoint

[tomquisel]

See the discussion in stellar/js-stellar-sdk#233.

Add the following fee preset suggestions to the response from /fee_stats as Tomer suggested:

Fee preset	Triggered at capacity	Percentile selected	Max Fee (stroops)
low	0.9	50th	10,000 (0.001 XLM)
med	0.7	70th	100,000 (0.01 XLM)
high	0.5	90th	1,000,000 (0.1 XLM)

This provides recommendations that are easy to use, and are implemented centrally rather than requiring reimplementation in each SDK.

When this is finished, be sure support for fetching these gets added to JS, Java, and Go SDKs.

[tomerweller]

Just to get on the same page:

• high fee recommendation should be used in user facing applications where interactions are very low frequency.
• low fee recommendation should be used by market making bots or any other scripts that submit transactions in high frequency.
• medium fee for anything in between.

And of course, all the information we use to calculate these numbers is readily available so consumers can choose any other combination.

@MonsieurNicolas can you take a look at the numbers above and?

[tomerweller]

Potential revised: /fee_stats (ignore the actual recommendation numbers, they're madeup):

{
  "last_ledger": "22658657",
  "last_ledger_base_fee": "100",
  "ledger_capacity_usage": "0.38",
  "min_accepted_fee": "100",
  "mode_accepted_fee": "100",
  "p10_accepted_fee": "100",
  "p20_accepted_fee": "100",
  "p30_accepted_fee": "100",
  "p40_accepted_fee": "100",
  "p50_accepted_fee": "100",
  "p60_accepted_fee": "100",
  "p70_accepted_fee": "100",
  "p80_accepted_fee": "100",
  "p90_accepted_fee": "275",
  "p95_accepted_fee": "475",
  "p99_accepted_fee": "8000",
  "fee_recommendations" : {
    "low" : 100,
    "medium" : 200,
    "high" : 475 
  }
}

I think the above creates sufficient sufficient separation between general stats and opinionated recommendations. But also open to a completely different endpoint. @tomquisel ?

[bartekn]

I think we shouldn't do it.

Normal users will (should) never interact with this endpoint directly and developers should take other things into consideration when setting fee (ex. timebounds.max_time on the transaction). Say, you set a low fee but the timeout of a transaction is 30 seconds, then it's possible that this transaction will never be added to the ledger.

Exposing opinionated values here will discourage developers from thinking more about how they should set the fees in their apps. They may also think that it's kind of a golden standard for fee selection but it's not.

[MonsieurNicolas]

Adding to @bartekn : I would stay clear of raw recommendations, provide data that allows people to take real decisions and make tradeoffs for their use case. Maybe we should be more explicit (by use of grouping), on the timelines for all these things:

{
  "last_ledger": "22658657",
  "last_ledger_base_fee": "100",
  "stats": {
    "ledger_count": 5,
    "ledger_capacity_usage": "0.38",
    "min_accepted_fee": "100",
    "p10_accepted_fee": "100",
    "p20_accepted_fee": "100",
    "p30_accepted_fee": "100",
    "p40_accepted_fee": "100",
    "p50_accepted_fee": "100",
    "p60_accepted_fee": "100",
    "p70_accepted_fee": "100",
    "p80_accepted_fee": "100",
    "p90_accepted_fee": "275",
    "p95_accepted_fee": "475",
    "p99_accepted_fee": "8000"
  }
}

• Is mode useful at all? If people compute fees, it seems unlikely that they will compute the same value.
• Maybe stats should be an array, and provide different ranges? Like last 5, last 1.
• Do we need that many percentiles on the top end? Seems to me that it's a bit backwards: people don't care about the highest bid, only care about the lowest bid, so if we're going to put more resolution it should be for the p5_accepted_fee. That said, maybe simplify it to only 10s (as min accepted fee is there as well) could be enough.
• for general stats, we could add a max_accepted_fee or a total_accepted_fees (but in general, maybe we don't want to turn this endpoint into that, and stick to data actually useful for computing bids).
• percentile names are not automation friendly and not future proof, maybe consider using a sorted array of tuples (people then look for the range containing what they want), like

percentiles: [
 [ 10, 100],
 [ 20, 100],
 [ 30, 100],
 [ 40, 100],
 [ 50, 100],
 [ 60, 100],
 [ 70, 100],
 [ 80, 100],
 [ 90, 275]
]

if we were flipping things around, the data would look something like this (truncated at 50th percentile):

percentiles: [
 [ 50, 275],
 [ 60, 100],
 [ 70, 100],
 [ 80, 100],
 [ 90, 100],
 [ 95, 100]
]

alternatively use a dictionary, but iterating over a dictionary when looking for ranges is not very natural.

[tomerweller]

I agree that experienced stellar developers should be mindful about fees and devise their own strategies. I also think that not all (probably most) developers fit in that bucket and by not suggesting strategies we're sending them to the extremes (min fee, 99th percentile, absurdly high numbers, etc.).

As a reference, ethgasstation is a common tool in the ethereum ecosystem for fee strategies.

@MonsieurNicolas regarding format and mode I tend to agree but this might fit better in a different issue.

[tomerweller]

@bartekn yeah we need to make sure it's clear in the api that these recommendations reflect the present and people should be mindful about transactions that will be submitted in the future.

[MonsieurNicolas]

OK, so I am starting to warm up to the idea of having values that are directly usable by clients (ie, clients just need simple tweaks to use them, like pad those values even more).

I think we need to dig deeper on this though:
I like that we can have a mapping for classes of transactions.

Using priorities "low", "medium", "high" can be a mechanism for that.

What they actually mean is what we should define better: using percentiles from past transactions as a way to define them is actually not useful as what people care about is chances of success for future ledgers without overbidding.

Maybe we should just map them to the criticality of the transactions and clearly explain the limitations so that we can change the algorithm used to compute those things without breaking the contract.

I think that the way to frame it is probably by attempting to expose the trade off being made: bid more to have a higher chance of getting included in the next 3 ledgers or not.

That said - there is something I am struggling with here:

• for wallets, I am not sure we actually need this, especially after CAP0005 goes live.
  • the number of transactions originated by humans should be much lower than the capacity of a ledger
  • the max bid for those should be order of magnitudes larger than the minimum needed (like $0.01)
• high value contracts (like payment channels), would use a custom strategy anyways.
• for trading bots, the bidding strategies are a lot more complicated as the failure rate is potentially a lot higher than what other uses cases are ok with. I would let bot SDK writers to figure this one out instead of trying to shoehorn bots into arbitrary buckets.