Recent fee rises and TxQ issues

[donovanhide]

During today's large increase in transaction fees, possibly due to a certain tweet, rippled is spewing lots of the following logs. Market makers connected to this instance are receiving "Can not queue at this time." responses. Is this the intended behaviour? The market makers are trying to use low transaction fees to get their offers onto the books and to make money by keeping the fees low. The end effect of the transaction fee increment process seems to be that the market makers are getting priced out of the market, which is ironic :-)

2017-Aug-28 22:47:21 TxQ:WRN Queue is full, and transaction C92AC1CBCF48EDB1FDBB790CCFBD6EA74F5BDE6ACE2D27352FB74B5EF472AD98 fee is lower than end item's account average fee
2017-Aug-28 22:47:21 TxQ:WRN Queue is full, and transaction 2C5AF5FB72BA96A8CBBA6980D98ED42C6105627856F96DC5269726F46DAB1586 fee is lower than end item's account average fee
2017-Aug-28 22:47:21 TxQ:WRN Queue is full, and transaction 1701E6EAC50CA0D9994AED3BECF49E14FA6DCDC000F175A202820A7376B80A3D would kick a transaction from the same account (rfepxFxQKMSwn1otQRk35TgXbfLX6p8JBX) out of the queue.
2017-Aug-28 22:47:22 TxQ:WRN Queue is full, and transaction 9DCDD2AF543932BD43954C20BD3365396A83AEF2F94339027C3DE572C6B8173E fee is lower than end item's account average fee
2017-Aug-28 22:47:22 TxQ:WRN Queue is full, and transaction F1F70AF66AEE4C497BB97522C6034BCEC3B3E8D92277D97D10F8BCB621142FC5 fee is lower than end item's account average fee
2017-Aug-28 22:47:23 TxQ:WRN Queue is full, and transaction 9696B9F9E3195423214B310B7F4885F0B90F032C4E57DCA4090BEC4F818682B7 fee is lower than end item's account average fee
2017-Aug-28 22:47:24 TxQ:WRN Removing last item of account rfepxFxQKMSwn1otQRk35TgXbfLX6p8JBXfrom queue with average fee of27297 in favor of 10982E55CC5219B0B4B90B5B2D205965196171F8C8A720800EF14D6B9F789C21 with fee of 446796
2017-Aug-28 22:47:24 TxQ:WRN Queue is full, and transaction A1DEDCCF06002577E40DA3C2302674C68957B0F479264B66056D4D1457CC2EFE fee is lower than end item's account average fee
2017-Aug-28 22:47:24 TxQ:WRN Queue is full, and transaction 7AB84471D805ADE13BE32F5E83506D14A35AF08FAD790AD8F4AAF244904A00C1 fee is lower than end item's account average fee
2017-Aug-28 22:47:24 TxQ:WRN Queue is full, and transaction E7EC553FB99FB7948005A09AA41AE94418D80109926E557B0A089571D39FEDBF fee is lower than end item's account average fee
2017-Aug-28 22:47:24 TxQ:WRN Queue is full, and transaction 688C1E363C3B9E8391BEA044A9D57E8B45CAD6CC90DB2992C3B205588D62E03E fee is lower than end item's account average fee
2017-Aug-28 22:47:24 TxQ:WRN Queue is full, and transaction 4361FE30EFBF320166B9289B8B4263D86D0381BD826D78A096DDE1E512F99E78 fee is lower than end item's account average fee
2017-Aug-28 22:47:26 TxQ:WRN Queue is full, and transaction C801A86FEA2707A656CA741DE727BD5D87F054F0F2830A5BE832245BE9A75B70 fee is lower than end item's account average fee
2017-Aug-28 22:47:26 TxQ:WRN Queue is full, and transaction 5FD6ED2C1B61AEC719E0CE0485E712AE4771B448F8112E09E9F4B648200A79E7 fee is lower than end item's account average fee
2017-Aug-28 22:47:30 TxQ:WRN Queue is full, and transaction AAE9D9E47BE9DD6B7B2648D277F3F9266C80224CE250042F846B0A1BFF5843EB fee is lower than end item's account average fee
2017-Aug-28 22:47:31 TxQ:WRN Queue is full, and transaction 22C24DBD28C4FAA2F36E584330A719C919D8FB0DCA57407613663D9933DE4540 fee is lower than end item's account average fee
2017-Aug-28 22:47:31 TxQ:WRN Queue is full, and transaction AD735AF3B1298BDF5B355FE922BA318977D5FEA2C851A68E8DF9E610C9C377A1 fee is lower than end item's account average fee
2017-Aug-28 22:47:31 TxQ:WRN Queue is full, and transaction 40B59A6B2F425ACFAD39CA4F1458031993B39DBDEEBBAF137138A57643B727F7 fee is lower than end item's account average fee
2017-Aug-28 22:47:32 TxQ:WRN Queue is full, and transaction A749A097EBD0EC4B2F7F47F0D1393102A332DF4369D6336E2225DE8815F568B7 fee is lower than end item's account average fee
2017-Aug-28 22:47:32 TxQ:WRN Removing last item of account rfepxFxQKMSwn1otQRk35TgXbfLX6p8JBXfrom queue with average fee of27469 in favor of 9C888D0ACA02C4D15AAE2253433EEF82B257C6B556AAC6BB99B9AA4E1DBF3F14 with fee of 38400
q2017-Aug-28 22:47:33 TxQ:WRN Queue is full, and transaction 6C0A59D1EB3E48494A30AEDFA6C85D3B6ABDABCA117B96DE67B344412551A353 fee is lower than end item's account average fee
2017-Aug-28 22:47:34 TxQ:WRN Queue is full, and transaction 50F231BEAC67BAB309FCBA3A19E2198A7183DC73339F34BD0D274AEF506E22F8 fee is lower than end item's account average fee
2017-Aug-28 22:47:34 TxQ:WRN Queue is full, and transaction 5DB0C39C7AA8F82D385D5A6B2709566989A18F2C83D86321D221BE8497F03F48 fee is lower than end item's account average fee
2017-Aug-28 22:47:35 TxQ:WRN Queue is full, and transaction 38B08A4E6CE70ABBE29053344069600C91A44BA9F8756AD4BB550874B603ABF6 fee is lower than end item's account average fee
2017-Aug-28 22:47:35 TxQ:WRN Queue is full, and transaction A32A8EF77A14D6E1039769DB11F86569D6857BE3662EC517F4168653C7BDD550 fee is lower than end item's account average fee
2017-Aug-28 22:47:36 TxQ:WRN Queue is full, and transaction C234E7971A60621DE677272426217020F656ACB14B305C2914E6DDD101C9A470 fee is lower than end item's account average fee
2017-Aug-28 22:47:36 TxQ:WRN Queue is full, and transaction 6BE17EAA6D8C93BBB1DCB4D39800B005BDCD44BF4C662AD0BBE9716852FD8B54 fee is lower than end item's account average fee
2017-Aug-28 22:47:37 TxQ:WRN Queue is full, and transaction 936F4E461768015C8F49C3297DAF4DC53EC628B9D6EBEE8BAE62A598CD076731 fee is lower than end item's account average fee
2017-Aug-28 22:47:37 TxQ:WRN Queue is full, and transaction B002C8025DA8A38029F45AB900B8017CABF655EE692323FFAB7063D15E684FC9 fee is lower than end item's account average fee
2017-Aug-28 22:47:37 TxQ:WRN Queue is full, and transaction 89C6E7F4C12A80EF58E063D72E863B044A04D2B995F0DB8FD5543E296F494A5D fee is lower than end item's account average fee
2017-Aug-28 22:47:38 TxQ:WRN Queue is full, and transaction 77658D10FE4E7285922EC58880D1C6BDA1D7C1FB16C8D303F6DBF7641FCE50B9 fee is lower than end item's account average fee```

[ximinez]

@donovanhide Thanks for the feedback and your question. As far as we can tell, yes, everything is processing exactly as designed. To prevent runaway resource usage and/or abuse, the queue was implemented with a size limit of 20 ledgers' worth of transactions (https://github.com/ripple/rippled/blob/develop/src/ripple/app/misc/FeeEscalation.md#other-constants).

If the network was staying healthy, and there were no problems with consensus times (and I haven't seen any evidence of problems yet), then it's safe to presume that the expected ledger size was at least 50, which means there were at least 1,000 transactions in the queue, which is honestly pretty impressive!

[JoelKatz]

It looks like that's the case. For example, here's a fee distribution from one of those ledgers. The first column is the number of transactions, the second is the fee paid in drops:

  4 10
  5 12
 13 30
  4 68
 12 70
  1 100
  2 490
  1 800
  1 5191
  9 10500

You'll notice that several 10 drop transactions still managed to get into the ledger and the total number of transactions in the ledger was 52.

[donovanhide]

Thanks for the responses. I guess the real question is how do I choose a fee which can make a profit based on the information fed out from rippled. For example here are two recent ledgers and the fee amounts advertised during the time before each ledger closes:

Ledger   | TxnCount | Max Fee | Fees
32299043 | 56       | 9700    | 5550,5739,5932,6128,6328,6530,6736,6945,7157,7372,7590,7812,8037,8265,8496,8730,8968,9209,9453,9700,10
32299042 | 36       | 8265    | 5363,5739,5932,6128,6328,6736,7372,7590,7812,8037,8265,5180,5363

32299043 seems to end on a fee of 10 drops, while 32299042 ends on 5363. I don't really know how to interpret this data. A market maker needs to be able to adjust his/her prices based on external data and/or trades occurring on the rippled ledger in a timely manner. Without being able to alter his/her offers in a timely manner, money will be lost or opportunities missed. If the fees are too high then profits will also decrease. Finding the balance to this problem is difficult, especially when the fees are sustained at such high levels as they were yesterday.

[ximinez]

I don't recognize the format of that data, but if those are all the fees that were paid by transactions on that ledger, it means that any transactions that paid less than those amounts were either put into the queue or kept in the queue.

Remember, that when a new ledger is opened, transactions are first pulled off the queue in order from highest fee to lowest. As those txs are added, the open ledger fee can start rising. And if it rises above the fee paid by the remaining highest fee tx in the queue, that tx will be left in the queue. Then both the open ledger and the queue are opened up for submissions. Transactions can be put in front of other transactions in the queue if they're willing to pay more. It's a trade-off between fee paid and time waited, and I have no idea how to make that decision for you.

Oh, there is one additional set of data points I can offer. If you use rippled's ledger command with ledger_index: current and queue: true (IIRC), you can see all of the transactions, including their fees, in both the open ledgers and the queue. I don't know if that'll help you make a decision for your own fees, but that's what we can give you.

[donovanhide]

The Fees values list is a result of this calculation based on the incoming ServerStream messages recorded from the notification of a ledger close until the next ledger close:

https://github.com/rubblelabs/ripple/blob/d26222abf520402bbd257f92f7ecdc6f5371a15a/websockets/subscribe.go#L47-L49

Based on this information:

https://ripple.com/build/transaction-cost/#server-state

The basic problem statement for a typical market-maker might be:
Every t seconds I want to submit n offers to replace existing offers from a set O which need to have their prices updated due to changed external data. These offers need to be updated within t seconds, otherwise the next batch will clash. I need to know what fee will accomplish this goal with a certainty greater than x percent. I can then use this value to adjust the offer prices accordingly.

I'd say that this is a usability problem. Currently, an understanding of a complex queue at the rippled end is required and the client needs to manage a further queue to dynamically feed the rippled queue. This is further complicated by the need to deal with cancellation of existing offers, which can get messy, as there is no way to mark an offer with a client-generated id, other than hacking the expires value.

If there was an estimateFee API call which could take parameters t,n and x and return a single fee value estimate, that would be very much more usable :-) Even better would be if this was a subscribe-able stream.

[donovanhide]

...
     {
            "LastLedgerSequence" : 32370764,
            "account" : "rG1dLiK6p4e6w3mvAVCEGAFv6TcbMN5wHY",
            "auth_change" : false,
            "fee" : "12",
            "fee_level" : "307",
            "max_spend_drops" : "12",
            "preflight_result" : "tesSUCCESS",
            "retries_remaining" : 10,
            "tx" : "F4011E477CA7EEE9ED3CBBD4D8A1768321578F4B6E6B76ED1DED6134A40DEA1A"
         },
...

Given the above section of the ledger current queue output, if that was my transaction, and I wanted to push it faster into a ledger, are you saying to re-sign it with a fee of 307 raised from the level of 12, and that would raise its chances closer to 100% of successful submission?

[ximinez]

https://github.com/ripple/rippled/blob/develop/src/ripple/app/misc/FeeEscalation.md#fee-level

[donovanhide]

Honestly, that link does not help or answer the question :-)

fee_level is 12/10*256

How do I use the information from the queue_data to help select a more appropriate fee?

[ximinez]

I think the crux of this problem you're describing is that you're asking rippled to predict the future actions of the other users / transactions / servers on the ripple network, and that's really just impossible without a well trained neural network or a time machine. All we can do is describe the past and present. If you want a (nearly) 100% chance of successful submission, the only real way is to pay the current open ledger fee (from the fee command, or using the load_factor from server_info). If you want to get into a given position in the queue, you can grab the ledger info as described above and pay a fee that will give you a fee level slightly higher than the transaction in that position, but there is absolutely no way I know of to predict or guarantee whether you'll stay in that position. The queue is essentially an auction, and you can't predict what others will bid.

[donovanhide]

I agree that a predictive statistical model of varying number of transactions combined with varying levels of fees yielding a chance of success output would be very useful, but it would certainly cost a large amount of XRP to produce the training data on the live network :-)

I'm more than happy to use the present values in rippled, which seems to be the open ledger fee. What I need to know is if the values in the server subscribe stream are appropriate. Strangely, I see the response is no longer documented here:

https://ripple.com/build/rippled-apis/#subscribe

except the brief excerpt:

Stream: server - Information about the server status, such as load_base (the current load level of the server), random (a randomly-generated value), and others, subject to change.

The formula used is here:

https://github.com/rubblelabs/ripple/blob/d26222abf520402bbd257f92f7ecdc6f5371a15a/websockets/subscribe.go#L35-L49

If I were to select the highest seen value after the last ledger close, and just before submitting my transactions, would that figure be a fair representation of the open ledger fee? Unfortunately there are four separate API responses where fee information is presented: server_state, server_info, server stream and fee and the docs are all a little vague about the fields in each.

[donovanhide]

Well, today it seems like:
rpcyXFq6ArWozyYLUE52FnNDjWWcnGYTGe
and
rDfdaugfiyePPcNKHL2soyL3wDmya32Ya3
are having fun paying huge fees and inflating the transaction fee for everyone else, resulting in hardly any transactions going through and servers reporting they are full. I'm sure they will probably run out of XRP and it's non-malicious, but it is very much observable that the transaction volume topples when the fee rises to current levels, which does suggest most bots do not have a suitable fee selection strategy, which goes back to the original point of this issue.

[gituser]

I wonder if it's the same issue that rippled when I do send some ripples gives me transaction hash today but when I look it at https://ripple.com/build/ripple-info-tool/ it says tx not found.

What do you guys think? I do not see any errors in rippled log though.

[donovanhide]

Ledger 32683368 is a good example of dysfunction. 4 out of 5 transactions all paying over 2 cents worth of XRP to be in an empty ledger, preceded by not particularly full ledgers.

[gituser]

All day today I'm getting 'transaction not found' on ripple-info-tool whilst locally transactions are accepted.

I've resent now few of them, but I'm worried that I might send two times.

[donovanhide]

@gituser You need to check the validated transactions up until your specified LastLedgerSequence. The response you get when you submit is effectively useless.

[gituser]

@donovanhide how do I do this?

[donovanhide]

@gituser Attempt to interpret this:

https://ripple.com/build/reliable-transaction-submission/

into your own code :-)

[gituser]

@donovanhide the thing is the method I've been using for sending was working perfectly fine until today. Is there some network issues at the moment?

I can see also via account_info that my account is "validated" : false

[donovanhide]

@gituser I can't really help you debug your code, but this issue is that today, and on previous days, when some accounts raise the overall transaction fee, most other bots seem to fail to get their transactions into a ledger.

[gituser]

@donovanhide ok, I'm looking into that documentation you sent before to make proper submission of the transaction in Ripple network.

Thank you.

[gituser]

@donovanhide so I read about reliable transaction posting and it seems just too much hassle to implement all of that and I'm not sure that even after I do create the system which will check for LastLedgerSequence and re-submit transactions it will work better and won't give me more headache. In current system the transaction is only sent once. And if something goes wrong you've got to re-send it manually, but this also protects from accidental double sending.

So here's workaround I've used - I've just increased and made fee static to 0.5 XRP instead of using fee_mult_max parameter after this all transactions seems to be mined much more reliable.

[tuloski]

Yeah the fees are getting "ridiculously" high for market makers...they are not low enough to not be considered in the risk model. And it seems there are not too many transactions per second. Are the validators overloaded by some "malicious" attack that don't claim any XRP and the load_factor goes high? How is the load_factor calculated?

[donovanhide]

./rippled -q json ledger '{"ledger_index":"current","queue":true}' | grep account | wc -l
366

Certainly lots of transactions in the queue, not many getting into ledgers...

[JoelKatz]

We've escalated and are investigating. We have definitely seen evidence of undesirable emergent behavior.

[donovanhide]

I'm seeing rpJrC46bPSUiWwTuuRGSiqHdSaCNWTFBZv consistently at the top of the queue with high fees. Oddly seems to lead to:

https://www.eobot.com/user/436753

Which suggests that maybe the ledger is being used for mining payouts and the payout bot has some bad fee calculation code?

Edit: seems that address is also linked to https://www.jubi.com/ ...

[tuloski]

What is the "latency" I see in rippled code to calculate if the server is overloaded? Where that time is coming from? I see there are 2ms removed for jitter so I suppose it is related to the communication with the other nodes, but it's weird that it is the only measure of load of the network. And then there are a lot of magic numbers (+8 *4) :)...hard to decode that code.

[tuloski]

The load factor I see is very high (minimum 3000 up to 400000 or more), so the fee is correctly calculated based on the loadFactor. Is there one of the validators having high load and then it propagates to the network?
s-west.ripple.com where I connect has a loadFactorServer = 1, so it might be another one.

[gituser]

No I don't. I'm getting tesSUCCESS for the new transactions.

The old ones of course I can try to find in the history, but I'm not running full rippled node, I've set pruning to 2000 there so old data is being deleted and also by looking via https://ripple.com/build/ripple-info-tool/ on all those transactions I'm getting Transaction not found.

EDIT: I've just checked the transaction which I sent earlier (about 1.5 hours ago) and it's not found anywhere neither on the tool neither in my local rippled instance.

[donovanhide]

You can imagine the Ops department in a bank phoning the dev late at night saying the exact same thing :-) There are definite usability questions about the transaction queue... Batch submission with atomic guarantees would go a long way to simplify the user experience.

My best advice is just wait for the fix; my market makers cannot operate at a profit with the way things are currently working.

[JoelKatz]

I can see also via account_info that my account is "validated" : false

That just means that you are asking the server for the most recent information, which is unvalidated. Most rippled queries default to querying the open ledger. Typically this gets you the most current information, taking into account even unverified transactions. That's usually what you want. (For example, if you submit a transaction to send XRP to someone else, you don't want to see that you still have the XRP just because the transaction isn't fully validated yet.)

Unfortunately, at the moment, open ledgers are diverging a bit and querying the open ledger can show you things that are less likely to correlate with future network states. Querying validated information is always safe unless you're querying a misconfigured server or the network's consensus logic as a whole is broken.

My best advice is just wait for the fix; my market makers cannot operate at a profit with the way things are currently working.

Anything relying on being able to clear large numbers of transactions at low cost should probably wait for a fix.

[gituser]

@JoelKatz,

I understand the difference between querying unvalidated vs validated open ledger.

The question is how to reliably submit new transaction?

1. I've added LastLedgerSequence which I get from server_state according to documentation and I'm adding to the resulting value 4.
2. If I omit Sequence in the transaction it defaults to the unvalidated open ledger (account_info without ledger:"validated") Sequence
3. If I specify Sequence in the transaction from the validated ledger (account_info with ledger: "validated") it gives me error tefPAST_SEQ

So what should be the logic regarding 2) & 3) if I submit let's say simultaneously (at the same time) multiple transactions will be there a problem?

Also, another question:
according to the documentation https://ripple.com/build/rippleapi/#transaction-fees there is maxFee parameter but If I specify maxFee the signing fails.

And another question:
How to replace some transaction with Sequence number? I do get on submit tefPAST_SEQ on my local rippled and transaction isn't going to the network.

@donovanhide

You can imagine the Ops department in a bank phoning the dev late at night saying the exact same thing :-) There are definite usability questions about the transaction queue... Batch submission with atomic guarantees would go a long way to simplify the user experience.

That's very true.

My best advice is just wait for the fix; my market makers cannot operate at a profit with the way things are currently working.

Seems it is the only way, but I'd like to understand more about how ripple works in terms of sending/confirming transactions and why some are confirmed faster than others.

[jnr101]

@tuloski,
I did some checks and wrote a very simple program only doing the api.getFee() for each ledger.
C1 = Client@home_local_VM
C2 = Client@remote_VM1
S1 = Rippled@wss://s1.ripple.com
S2 = Rippled@remote_VM2

I tried all combinations. It turns out that each rippled gives different fees back. (not sure if that is worriesome, I can imagine it will also look at local load). When connecting to the same rippled it will give the same fees back. It was at first a bit confusing, but even when connecting to s1.ripple.com it sometimes gives different fees. This is because there are several rippleds behind that address. The factor of difference can be significant. Below are 2 connections to s1.ripple.com at the same time:

[15/09/2017 21:15:03] Connected..
[15/09/2017 21:15:06] Fee (0.10004165625)
[15/09/2017 21:15:09] Fee (0.10004165625)
[15/09/2017 21:15:13] Fee (0.098415)
[15/09/2017 21:15:17] Fee (0.098415)
[15/09/2017 21:15:21] Fee (0.10086000000000002)
[15/09/2017 21:15:24] Fee (0.097606640625)
[15/09/2017 21:15:27] Fee (0.097606640625)
[15/09/2017 21:15:30] Fee (0.097606640625)
[15/09/2017 21:15:33] Fee (0.098415)
[15/09/2017 21:15:38] Fee (0.099226640625)

[15/09/2017 21:14:54] Connected..
[15/09/2017 21:14:57] Fee (0.152006625)
[15/09/2017 21:15:01] Fee (0.15)
[15/09/2017 21:15:05] Fee (0.151001625)
[15/09/2017 21:15:09] Fee (0.152006625)
[15/09/2017 21:15:12] Fee (0.15)
[15/09/2017 21:15:17] Fee (0.15)
[15/09/2017 21:15:20] Fee (0.152006625)
[15/09/2017 21:15:23] Fee (0.14900165625)
[15/09/2017 21:15:26] Fee (0.14900165625)
[15/09/2017 21:15:29] Fee (0.14900165625)
[15/09/2017 21:15:33] Fee (0.15)
[15/09/2017 21:15:38] Fee (0.151001625)

The client (C1 or C2) do not make the difference in fees.

[jnr101]

could it be that the systems are busy due to increasing use of some of the new functionality, which then would use more system resources then perhaps anticipated? in that case it would be just organic growth and the fee-escalation should be loosened somewhat. Or maybe we're dealing with non-optimal implementation of the new functionality..(But I am just wildly guessing here of course)

[tuloski]

@gituser I'm using the MaxFee without problem...just the transactions don't go through because my max fee is often not enough and it fails with telINSUF_FEE_P

@jnr101 I don't get why 2 different rippled should give different fees. At the end the fee that matters is the open ledger, because you could modify your rippled code to "ask" you 0 fees, but then it would fail in the consensus. In your case your rippled was giving a lower fee and then the transaction was validated? So now I can say that whatever I thought to know about the fees is not true :)

[gituser]

@tuloski

I'm using the MaxFee without problem...just the transactions don't go through because my max fee is often not enough and it fails with telINSUF_FEE_P

I'm getting this when I try to sign transaction with MaxFee field inside tx_json structure:
Field "tx_json.MaxFee" is unknown.

[jnr101]

@tuloski All I can say is my transactions validate with apparently a lower fee setting than what @gituser is using. I don't know why. But I can say that my instance of rippled is configured 'tiny' and running on relatively fast system. And perhaps of influence, I do not allow inbound connections on my rippled

You can try the test with the fees with this code:

'use strict'

const {RippleAPI} = require('ripple-lib')
var moment = require('moment')

const feeCushion = 1.2

const api = new RippleAPI({
  feeCushion: feeCushion,
  server: 'wss://s1.ripple.com' // Public rippled server hosted by Ripple, Inc.
});

function timestamp() {
  return moment().format('DD/MM/YYYY HH:mm:ss')
}

api.on('error', (errorCode, errorMessage) => {
  console.log(errorCode + ': ' + errorMessage);
});

api.on('ledger', ledger => {
  return api.getFee().then(fee => {
      console.log("[" + timestamp() + "] Fee (" + fee + ")")
  }).catch(console.error);
});

api.on('error', (errorCode, errorMessage, data) => {
  console.log(errorCode + ': ' + errorMessage);
});

api.connect().then(() => {
  console.log("[" + timestamp() + "] Connected..");
}).catch(console.error);

install/run with

npm install moment ripple-lib
node getFee.js

start two instances and when it connects to different server you will find different fees

For comparison here are some fee results from my own rippled:

[15/09/2017 22:08:05] Connected..
[15/09/2017 22:08:07] Fee (0.06402665625000001)
[15/09/2017 22:08:11] Fee (0.06272662500000001)
[15/09/2017 22:08:15] Fee (0.063375)
[15/09/2017 22:08:19] Fee (0.063375)
[15/09/2017 22:08:23] Fee (0.06402665625000001)
[15/09/2017 22:08:26] Fee (0.062081625)
[15/09/2017 22:08:29] Fee (0.063375)
[15/09/2017 22:08:33] Fee (0.5360574375)
[15/09/2017 22:08:36] Fee (0.06402665625000001)
[15/09/2017 22:08:39] Fee (0.062081625)

[gituser]

Regarding: Sequence number issue it seems to be a regression in the v0.70.1, I've switched back to the v0.70.0 and a miracle - now the Sequence for both validated/unvalidated account_info is the same and I also can send now transaction with previous Sequence!

EDIT: hm, no, after some further testing there is a difference again between Sequence numbers for validated / unvalidated account_info even on v0.70.0.

[wilsonianb]

@donovanhide do you have maximum_txn_per_account specified in your config to something greater than 366? I'm not seeing any accounts with more than the default 10 queued transactions.
https://github.com/ripple/rippled/blob/develop/doc/rippled-example.cfg#L535-L538

[donovanhide]

@wilsonianb

Yep, had it whacked up since Ed advised me to do so about a year ago :-) I submit batches of market maker transactions greater than 10 about once a minute:

$ grep maximum_txn_per_account rippled.cfg
maximum_txn_per_account=1000

[donovanhide]

I'm now seeing less transactions per ledger than a Chinese Bitcoin Exchange circa 1st October 2017 :-)

[tuloski]

BTW I'm seeing many transactions with low fees (0.000011) getting into the ledger. I don't get how is that possible...they might have their local rippled with low load and low fee, but then why don't they fail in the consensus? How can I retrieve the open ledger fee?
I see mainly these accounts with very low fee: rUp8CvtneUUL6qYaxfZwUHg2cwYuo5jugG, rEp4SYVfwQgGbWCJV4tBAgGNpG2KeiaY1W, rD8LigXE7165r3VWhSQ4FwzJy7PNrTMwUq

[donovanhide]

Hi Ed (@ximinez)!

thought I'd give the 0.70.2 hotfix an early quick spin. The queue submission responses are a lot more enlightening than the old blanket telINSUF_FEE_P, which is great :-)

So, my market making bots now can submit transactions to the queue, and now strangely, exactly 10 from each batch always seems to succeed and the rest get queued locally, but do not succeed before the next batch, roughly a minute later. Is this the maximum_txn_per_account setting being applied at the validator end? Maybe I'm being premature, and 0.70.2 need to be deployed elsewhere :-)

[ximinez]

Is this the maximum_txn_per_account setting being applied at the validator end?

Most likely.

[JoelKatz]

We don't expect the fixes in 0.70.2 to have much effect until they're more widely deployed. We are still testing and finalizing the build.

[donovanhide]

Queues have been drained and local load_factor seems to be back to normal. Guess the fix has been deployed to the validators. Any chance of a tagged release?

[ximinez]

Soon™️

[ximinez]

0.70.2 has been released to master and tagged. Enjoy!

I'm going to go ahead and close this issue. If you have any further problems, feel free to reopen or open a new issue.

[donovanhide]

Nice work. Trading has resumed :-)

If you do get a spare moment after this slog, would be good to get some thoughts on building a local model for predicting a transaction fee for a range of possible transaction numbers in the next ledger:

https://www.xrpchat.com/topic/9809-questions-on-fees/?do=findComment&comment=98593