Horizon timeout after transaction is added to ledger #305

oryband · 2018-02-12T16:57:25Z

See the following transaction:
https://horizon.stellar.org/transactions/335b869b7c9eb491291869b8b06b32582190395945cccb91457ba04774f3ace0
This transaction was submitted to Horizon on 2018-02-12T15:12:19.31094599Z. Looking at the transaction information, it was created on 2018-02-12T15:12:19Z.
Transaction added to ledger on: 2018-02-12T15:12:19Z (~5 seconds after submission): https://horizon.stellar.org/ledgers/16230126
Horizon replied with an HTTP 504 (Gateway Timeout) on 2018-02-12T15:13:19.434419221Z (~1 minute after submission).
I've used the golang SDK, and my horizon.Client.HTTP.Timeout is set to 2 minutes.

This is reproducible on 4 identical machines on DigitalOcean. The instances are on DO's zones FRA1, SFO1, SGP1, NYC1

Each node is running Horizon + Core (watcher, non-validating) using stellar/quickstart docker image with this commit: stellar/quickstart@aad0807

A local application submits transactions to the local Horizon on each node.

Here is the following Core and Horizon configuration in use:

# stellar/quickstart specific
HTTP_PORT=11626
PUBLIC_HTTP_PORT=true
LOG_FILE_PATH=""
DATABASE="postgresql://dbname=core host=localhost user=stellar password=1"
CATCHUP_RECENT=1024

NETWORK_PASSPHRASE="Public Global Stellar Network ; September 2015"

NODE_NAMES=[
"GC5SXLNAM3C4NMGK2PXK4R34B5GNZ47FYQ24ZIBFDFOCU6D4KBN4POAE  satoshipay1",
"GBJQUIXUO4XSNPAUT6ODLZUJRV2NPXYASKUBY4G5MYP3M47PCVI55MNT  satoshipay2",
"GAK6Z5UVGUVSEK6PEOCAYJISTT5EJBB34PN3NOLEQG2SUKXRVV2F6HZY  satoshipay3",
"GCGWABAQ6OUOVUGWJVPRJ5LWBIWYN3CVOVOZYBNQQGIBRULQHYNGQ7GH  cryptomover1",
"GC7MH45NSXXPBLQJRSEVF2DFUVLGGYOJER5FRUNVCYVMXJYJT5LLQJW5  cryptomover2",
"GAENPO2XRTTMAJXDWM3E3GAALNLG4HVMKJ4QF525TR25RI42YPEDULOW  ibm_uk",
"GBJ7T3BTLX2BP3T5Q4256PUF7JMDAB35LLO32QRDYE67TDDMN7H33GGE  ibm_hong_kong",
"GCGB2S2KGYARPVIA37HYZXVRM2YZUEXA6S33ZU5BUDC6THSB62LZSTYH  sdf1",
"GCM6QMP3DLRPTAZW2UZPCPX2LF3SXWXKPMP3GKFZBDSF3QZGV2G5QSTK  sdf2",
"GABMKJM6I25XI4K7U6XWMULOUQIQ27BCTMLS6BYYSOWKTBUXVRJSXHYQ  sdf3",
]

KNOWN_PEERS=[
"core-live-a.stellar.org:11625",
"core-live-b.stellar.org:11625",
"core-live-c.stellar.org:11625",
"confucius.strllar.org",
#"stellar1.bitventure.co",
"stellar.256kw.com"
]

[QUORUM_SET]
#VALIDATORS=[
#"$eno", "$tempo.eu.com", "$satoshipay", "$cryptomover", "$umbrel", "$exodo", "$ibm", "$sdf_watcher1", "$sdf_watcher2", "$sdf_watcher3"
#]
VALIDATORS=[
"$sdf1", "$sdf2", "$sdf3", "$satoshipay1", "$satoshipay2", "$satoshipay3", "$cryptomover1", "$cryptomover2", "$ibm_uk", "$ibm_hong_kong"
]


[HISTORY.cache]
get="cp /opt/stellar/history-cache/{0} {1}"

[HISTORY.sdf1]
get="curl -sf http://history.stellar.org/prd/core-live/core_live_001/{0} -o {1}"

[HISTORY.sdf2]
get="curl -sf http://history.stellar.org/prd/core-live/core_live_002/{0} -o {1}"

[HISTORY.sdf3]
get="curl -sf http://history.stellar.org/prd/core-live/core_live_003/{0} -o {1}"

#!/bin/bash

export DATABASE_URL="postgres://stellar:1@localhost/horizon"
export STELLAR_CORE_DATABASE_URL="postgres://stellar:1@localhost/core"
export STELLAR_CORE_URL="http://localhost:11626"
export LOG_LEVEL="info"
export INGEST="true"
export PER_HOUR_RATE_LIMIT="72000"
export NETWORK_PASSPHRASE="Public Global Stellar Network ; September 2015"

It looks like the configuration is OK, no idea why I'm getting the timeouts.

The text was updated successfully, but these errors were encountered:

oryband · 2018-02-12T17:22:02Z

A node that succeeds submitting transaction returns the following /metrics response:

"txsub.buffered": {
  "value": 0
// ...
"txsub.open": {
  "value": 11

Nodes that are timing out return the following:

"txsub.buffered": {
  "value": 0
// ...
"txsub.open": {
  "value": 58

oryband · 2018-02-13T10:29:55Z

Possibly related issues:

504 (timeout) when submitting a transaction to horizon stellar-deprecated/horizon#369
Horizon timeouts when submitting tx in dev environment stellar-deprecated/horizon#354

oryband · 2018-02-14T11:04:19Z

I've managed to bypass the issue by not using the quickstart docker image. I created an alternative setup similar to the above, but on AWS. At first I tried to used the quickstart docker image but got the same results - not a single successful transaction, all timed out.

Afterwards I decided to abandon the docker image and setup my ow PostgreSQL on RDS, and four EC2 instances in four different regions globally, and the problem has disappeared completely.

bartekn · 2018-02-14T19:21:55Z

Moving to stellar/quickstart#41.

s-a-y · 2018-02-26T12:20:12Z

Have the same issue, but not using quickstart docker image.
Using that https://hub.docker.com/r/otokarev/stellar-horizon/~/dockerfile/

andrenarchy · 2018-07-09T21:35:18Z

It doesn't look like this is related to the docker image (I can imagine it's not related to docker at all). We currently have one Horizon that is timing out with 504s for all transactions that are submitted to it. Anything I can do to help debugging this?

andrenarchy · 2018-07-09T21:41:33Z

Hm, I just set LOG_LEVEL=debug and after restarting Horizon the 504s disappeared. We keep getting this issue frequently – any idea what we can do next time it happens?

bartekn · 2018-07-12T14:41:56Z

@andrenarchy what's the average tx submission rate to all your Horizon instances? Does it happen when you submit a lot of transactions or even for smaller numbers?

bartekn · 2018-07-12T16:05:26Z

I think I'm able to reproduce it, going to debug it now.

andrenarchy · 2018-07-12T20:08:12Z

@bartekn cool, keep me posted! When Horizon was in this state it was enough to submit a single transaction to get the 504. The instance might have seen a large tx submission rate before though.

bartekn · 2018-07-16T15:56:51Z

After checking the txsub package in Horizon and running several tests I think that the issue discussed here is not a bug in Horizon but a lack of proper documentation for Timeout errors.

First, let me explain how txsub package work (simplified): when it sees a new transaction sent via POST /transactions endpoint it is forwarding it to the Core's /tx endpoint that is responsible for checking and then propagating a new transaction to the network. If Horizon sees PENDING response it starts monitoring txhistory table of the Core DB and if it can't see a newly submitted transaction for SubmissionTimeout (currently equal 1 minute) it sends Timeout error response.

Now, why Core doesn't add a new transaction to txhistory? This table contains only the transactions that were included in the ledger and if a new transaction is not included in 3 consecutive ledgers it's removed from the queue. This can happen when ledgers are full and network can't add extra transactions above max_tx_set_size (currently 50 in Stellar Public Network).

So what's the workaround in case of such situation? Developers have two options:

You can keep resubmitting the transaction (with the same sequence number) until it is included in the ledger.
You can increase the fee.

Core team is discussing a new fee model in stellar/stellar-protocol#133 that's basically changing fee to max_fee. This will make submitting transactions cheaper in a future (higher fee only when needed).

This issue will be closed after merging #541 which updates the documentation.

andrenarchy · 2018-07-17T07:53:29Z

Thanks for the explanation @bartekn!

However, this does not explain the issue for me. We observe that the transaction is actually included pretty much immediately after submitting it to Horizon (usually the next ledger). This is also what @oryband described above. Thus I don't see how the window of 3 consecutive ledgers in txhistory explains the behavior.

@bartekn can you please reopen the issue?

bartekn · 2018-07-17T07:59:50Z

@andrenarchy can you confirm that response body you're getting is sent by Horizon, i.e. it looks like this: https://github.com/stellar/go/blob/master/services/horizon/internal/render/problem/main.go#L64 Are you using any reverse proxy or communicate with Horizon directly?

Thus I don't see how the window of 3 consecutive ledgers in txhistory explains the behavior.

I will confirm with the Core team if this behaviour has been changed recently.

andrenarchy · 2018-07-17T08:24:46Z

We are using a reverse proxy but the timeout is not coming from the proxy. I can't reproduce it right now (Horizon is only showing this behavior sometimes) but I assume the response looks like in the link you sent.

Just to be clear: once Horizon is in this state it times out for all transactions sent to it – also when there are less than 50 tx per ledger.

@bartekn how did you manage to reproduce it? I'm happy to help debugging. :)

s-a-y · 2018-08-28T03:34:21Z

Yes, issue I had is not caused by high load.
When it happens - all transaction are getting successfully included into ledger, but Horizon always returns 504. I mean I wasn't able to get 200 response at all for any posted tx.
I believe it has something to do with misconfigured communication between horizon and stellar-core. I was able to overcome it when switched to different docker setup, so never really needed to dig deeper.
I wasn't using quickstart image.

ramisra · 2018-08-28T08:03:37Z

Yes @s-a-y this is not caused on high load as the transaction which I am submitting to horizon server is much low than 50 TPS and moreover the transaction gets included to ledger but still the horizon gets timeout.

@bartekn can you check this and need to reopen this issue.

bartekn · 2018-08-29T15:53:00Z

OK, it's possible that it's some obscure sync issue (like #603). Will try to debug it again.

gituser · 2018-09-11T21:12:01Z

Hi.

So what's up with this issue? Has it been fixed? Or what is a proper workaround how to deal with horizon if it returns 504 error? How to make sure that the transaction has been submitted and get it's resulting hash?

We've just stumbled upon it and don't know how to resolve it and get proper response from horizon that transaction has been successfully submitted.

And yes there is absolutely 0 load on our host. We only submit our own transactions from 1 account.

Thanks.

i3bitcoin · 2018-09-14T18:02:36Z

@gituser
Try to change in horizon.env this row
export LOG_LEVEL="info"
to
export LOG_LEVEL="debug"

It worked for me.

gituser · 2018-09-14T18:56:03Z

@gituser
Try to change in horizon.env this row
export LOG_LEVEL="info"
to
export LOG_LEVEL="debug"

It worked for me.

What LOG_LEVEL has to do with timeouts? I've just checked I had already LOG_LEVEL="debug" so the issue is with something else.

bartekn · 2018-10-19T11:44:43Z

Related stellar-core issue: stellar/stellar-core#1811

gituser · 2018-12-02T12:58:12Z

I just had again this error on fresh v0.15.1 stellar-horizon version (I'm running my own node):

transferMoney returned in exception Server error: `POST http://x.x.x.x:8000/transactions` resulted in a `504 Gateway Timeout` response: { "type": "https://stellar.org/horizon-errors/timeout", "title": "Timeout", "status": 504, "detail": "Your reque (truncated...)

Transaction of course has been included in the ledger.

gituser · 2018-12-02T19:14:21Z

I had to rollback to v0.14.2 because v0.15.1 is giving me 504 error when I try to send a transaction.

slavkomae · 2018-12-04T05:39:01Z

I'm consistently encountering this problem for each transaction I try posting on 0.14.2/10.0.0. Have to resort to using SDF servers. The transactions do actually materialize, but the response is 504.

gituser · 2018-12-04T08:21:20Z

@slavkomae interestingly I have this issue only on newer v0.15.1 horizon and on v0.14.2 it works for me.

bartekn · 2018-12-06T18:46:18Z

The txsub code did not really change between 0.14.2 and 0.15.1. When you receive Timeout response you should continue as described in: https://www.stellar.org/developers/horizon/reference/errors/timeout.html.

If you are constantly having this issue it means that your stellar-core has problems with communicating with the network (ie. transaction submission will not work if your node is not synced correctly) or the network is congested at the moment and you just need to try again.

If you want to receive a final state of a transaction in a defined time please add a timebound.max_time on your transaction. We're in the process of updating SDKs to give you a better control of timeouts. Java SDK 0.4.0 already has a support for this, other SDKs will follow.

I am going to close this as it's not Horizon issue - Horizon just forwards the transaction to stellar-core and waits for results. It has no control over how the transaction is broadcasted in the network.

oryband · 2018-12-23T15:04:56Z

temporary solution that worked for me: stellar/quickstart#41 (comment)

gituser · 2018-12-28T15:50:27Z

The txsub code did not really change between 0.14.2 and 0.15.1. When you receive Timeout response you should continue as described in: https://www.stellar.org/developers/horizon/reference/errors/timeout.html.

If you are constantly having this issue it means that your stellar-core has problems with communicating with the network (ie. transaction submission will not work if your node is not synced correctly) or the network is congested at the moment and you just need to try again.

If you want to receive a final state of a transaction in a defined time please add a timebound.max_time on your transaction. We're in the process of updating SDKs to give you a better control of timeouts. Java SDK 0.4.0 already has a support for this, other SDKs will follow.

The issue has nothing to do with timeout. The transaction gets included in 1 second in the ledger and transaction time indicates that, so the question is why horizon responds with 504 whilst in fact the transaction has been included in 1 second right after posting?

For me issue occurs every time I try to send outbound transaction through horizon on a fully synced node. What could be an issue there? My guess is after reboot there is gap in between horizon and stellar-core so it might be that it's trying to get some ledgers that are not available anymore because they were pruned? Althrough my instance shows that it's fully synced (right now I'm using latest versions v0.10.1 stellar-core and horizon-v0.15.2):

./stellar_status.sh 
=============== Horizon Status ========================

{
  "_links": {
    "self": {
      "href": "/metrics"
    }
  },
  "goroutines": {
    "value": 14
  },
  "history.elder_ledger": {
    "value": 21648592
  },
  "history.latest_ledger": {
    "value": 21698591
  },
  "history.open_connections": {
    "value": 3
  },
  "ingester.clear_ledger": {
    "15m.rate": 0,
    "1m.rate": 0,
    "5m.rate": 0,
    "75%": 0,
    "95%": 0,
    "99%": 0,
    "99.9%": 0,
    "count": 0,
    "max": 0,
    "mean": 0,
    "mean.rate": 0,
    "median": 0,
    "min": 0,
    "stddev": 0
  },
  "ingester.ingest_ledger": {
    "15m.rate": 0.19819669337980636,
    "1m.rate": 0.19404578929940278,
    "5m.rate": 0.19719863227960607,
    "75%": 8850709.25,
    "95%": 24417130.699999988,
    "99%": 36502417.700000055,
    "99.9%": 67616060.21900001,
    "count": 3220,
    "max": 67680988,
    "mean": 7554485.706225681,
    "mean.rate": 0.19646135932799777,
    "median": 4629571.5,
    "min": 13587,
    "stddev": 7813484.569750753
  },
  "logging.debug": {
    "15m.rate": 21.979505437210594,
    "1m.rate": 20.668846668958178,
    "5m.rate": 21.720380417132013,
    "count": 337123,
    "mean.rate": 20.568631674801082
  },
  "logging.error": {
    "15m.rate": 0,
    "1m.rate": 0,
    "5m.rate": 0,
    "count": 0,
    "mean.rate": 0
  },
  "logging.info": {
    "15m.rate": 2.4888746769642096,
    "1m.rate": 2.496900466047106,
    "5m.rate": 2.503651115522853,
    "count": 40390,
    "mean.rate": 2.464284654009363
  },
  "logging.panic": {
    "15m.rate": 0,
    "1m.rate": 0,
    "5m.rate": 0,
    "count": 0,
    "mean.rate": 0
  },
  "logging.warning": {
    "15m.rate": 0,
    "1m.rate": 0,
    "5m.rate": 0,
    "count": 0,
    "mean.rate": 0
  },
  "requests.failed": {
    "15m.rate": 0.0008581541516812897,
    "1m.rate": 0.00034600572835188285,
    "5m.rate": 0.0015356970441238928,
    "count": 1,
    "mean.rate": 0.00006101281911958095
  },
  "requests.succeeded": {
    "15m.rate": 0.04538249223267948,
    "1m.rate": 0.054058437995798514,
    "5m.rate": 0.0530912284376962,
    "count": 583,
    "mean.rate": 0.03557047354642922
  },
  "requests.total": {
    "15m.rate": 0.046240646384360776,
    "1m.rate": 0.054404443724150396,
    "5m.rate": 0.05462692548182013,
    "75%": 3893907.25,
    "95%": 5649787.25,
    "99%": 7383244.549999999,
    "99.9%": 39268929,
    "count": 584,
    "max": 39268929,
    "mean": 3000574.0684931506,
    "mean.rate": 0.03563148636549179,
    "median": 2148052.5,
    "min": 91263,
    "stddev": 2418799.11778822
  },
  "stellar_core.latest_ledger": {
    "value": 21698592
  },
  "stellar_core.open_connections": {
    "value": 2
  },
  "txsub.buffered": {
    "value": 0
  },
  "txsub.failed": {
    "15m.rate": 0,
    "1m.rate": 0,
    "5m.rate": 0,
    "count": 0,
    "mean.rate": 0
  },
  "txsub.open": {
    "value": 0
  },
  "txsub.succeeded": {
    "15m.rate": 0,
    "1m.rate": 0,
    "5m.rate": 0,
    "count": 0,
    "mean.rate": 0
  },
  "txsub.total": {
    "15m.rate": 0,
    "1m.rate": 0,
    "5m.rate": 0,
    "75%": 0,
    "95%": 0,
    "99%": 0,
    "99.9%": 0,
    "count": 0,
    "max": 0,
    "mean": 0,
    "mean.rate": 0,
    "median": 0,
    "min": 0,
    "stddev": 0
  }
}
=====================================================

=============== Stellar Status ========================

Using DEPRECATED command-line syntax.
Please refer to documentation for new syntax.

Content-Length: 1141
Content-Type: application/json

2018-12-28T18:52:06.713 GAZBQ [default INFO] {
   "info" : {
      "build" : "v10.1.0",
      "history" : {
         "sdf1" : {
            "failure" : 0,
            "success" : 0
         },
         "sdf2" : {
            "failure" : 0,
            "success" : 0
         },
         "sdf3" : {
            "failure" : 0,
            "success" : 0
         }
      },
      "ledger" : {
         "age" : 1,
         "baseFee" : 100,
         "baseReserve" : 5000000,
         "closeTime" : 1546012325,
         "hash" : "3a0b15b9bd8e9756ddf1220d3adb3ccef89afd8cae882b7de92ed896c1b41ae6",
         "maxTxSetSize" : 50,
         "num" : 21698592,
         "version" : 10
      },
      "network" : "Public Global Stellar Network ; September 2015",
      "peers" : {
         "authenticated_count" : 8,
         "pending_count" : 0
      },
      "protocol_version" : 10,
      "quorum" : {
         "21698592" : {
            "agree" : 6,
            "disagree" : 0,
            "fail_at" : 2,
            "hash" : "d5c247",
            "missing" : 1,
            "phase" : "EXTERNALIZE"
         }
      },
      "startedOn" : "2018-12-28T11:08:58Z",
      "state" : "Synced!"
   }
}


=====================================================

I am going to close this as it's not Horizon issue - Horizon just forwards the transaction to stellar-core and waits for results. It has no control over how the transaction is broadcasted in the network.

Of course, but horizon is used to send this transaction to the stellar network so for many applications this is an issue right there. Something timeouts means transaction wasn't successful but in fact it was.

gituser · 2018-12-29T23:21:56Z

temporary solution that worked for me: stellar/docker-stellar-core-horizon#41 (comment)

Indeed this works in LXC container as well.

oryband · 2019-01-17T10:04:01Z

Please note this comment stellar/quickstart#41 (comment)

bartekn · 2019-01-17T12:35:45Z

@oryband thanks! I've received a bunch of logs from Lumenauts and after analyzing them I think I found the issue (#815). It's now in review and will be released later today.

oryband · 2019-01-17T13:33:17Z

@bartekn looks like the PR solves the issue, I have an environment where I can reproduce this and it doesn't appear anymore when testing with the above branch. See my comment on the PR.

bartekn · 2019-01-17T19:34:50Z

Horizon 0.15.4 contains a fix for this issue. Can others confirm it's working for you?

andrenarchy · 2019-01-17T20:47:44Z

The SatoshiPay docker images have been updated to 0.15.4 and things seem to run smoothly with them. We haven't seen this issue very often though so it's hard to tell – maybe there's someone who saw that problem more frequently with the SatoshiPay images and who wants to give the new version a shot?

@bartekn thx for the fix, great job! :)

s-a-y · 2019-01-17T21:19:31Z

I'll update my images and have a look. Thanks a lot!

oryband · 2019-01-17T21:26:38Z

maybe there's someone who saw that problem more frequently with the SatoshiPay images

it's reproducable if not using the stellar testnet passphrase, and horizon launches before core is available for it. submitting a transaction in this case will timeout with HTTP 504.

gituser · 2019-01-18T19:40:37Z

Yes, seems that v0.15.4 fixes the issue with 504 error.

Also, I've added this to the systemd units (just to be sure):

/etc/systemd/system/stellar-core.service

[Unit]
Description=SDF - stellar-core
Before=multi-user.target
Requires=postgresql.service
After=postgresql.service

[Service]
User=stellar
Group=stellar
ExecStart=/home/stellar/stellar-core --conf /home/stellar/stellar-core.cfg
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
RestartPreventExitStatus=255
Type=simple

[Install]
WantedBy=multi-user.target

/etc/systemd/system/horizon.service

[Unit]
Description=SDF - stellar-horizon
Before=multi-user.target
Requires=postgresql.service stellar-core.service
After=postgresql.service stellar-core.service

[Service]
User=stellar
Group=stellar
LimitNOFILE=8192
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=stellar-horizon
EnvironmentFile=/home/stellar/.horizon-env
ExecStart=/home/stellar/horizon
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
RestartPreventExitStatus=255
Type=simple

[Install]
WantedBy=multi-user.target

TFiroozian · 2019-11-06T20:13:21Z

@bartekn We've experienced this issue after submitting a transaction with version 0.22.1 of the horizon many times.

Invalid horizon reply: [504] {'type': 'https://stellar.org/horizon-errors/timeout', 'title': 'Timeout', 'status': 504, 'detail': 'Your request timed out before completing. Please try your request again. If you are submitting a transaction make sure you are sending exactly the same transaction (with the same sequence number).'}

Also, I don’t think this is just a TX prioritization issue caused by node/network volume.
can you please check this and re-open this issue.

ire-and-curses · 2019-11-11T17:35:56Z

@TFiroozian It's not clear to me that what you're reporting is the same issue, since others reported success after the fix in 0.15. Could you open a new one and reference this one? Thanks.

gituser · 2021-01-07T21:26:41Z

It seems this issue is still there.

It happens from time to time: usually when there are lots of transactions being made in Stellar network.

Happens on:

stellar-core v15.1.0
horizon v1.13.0

Not sure how to debug it.

ire-and-curses · 2021-01-07T23:46:19Z

@gituser Note that it is normal and expected to receive 504 errors when the network is under heavy load, if your provided maximum fee is insufficient. Please confirm the hash of the failed transaction, the output of /fee_stats at the time, and the fee you used. If you still feel the behaviour is incorrect, please open a new issue and include all of that information and we can take a look.

gituser · 2021-01-08T00:23:22Z

@ire-and-curses the thing is that transaction is being included in the ledger 30 seconds later after the request.

Is there any way to get optimal fee rate for current network condition? I think I'm using standard stellar fee.

The error is still weird to me, should be more detailed.

Also I couldn't find anywhere where the timeout is defined for POST /transactions request to the horizon.

ire-and-curses · 2021-01-08T21:30:44Z

@gituser The general best practice is as follows:

Set a timebound for your transaction. This guarantees that after that time the transaction won't be accepted by the network. This protects you from e.g. sending a second duplicate payment.
Set an appropriate fee. If you like, you can get detailed information on the current necessary fees by querying the fee_stats endpoint, however...
...Set the highest maximum fee you are comfortable with. This doesn't mean you pay that in full on every transaction! You will only pay whatever is necessary to get you into the ledger. So under normal circumstances, even with a higher max fee set, you will pay the standard fee (currently 100 stroops).
Implement a retry loop with increasing delay (e.g. 30s, 60s, 90s). This should only execute once you've exceeded the timebound you set.
The timeout error happens because Horizon cannot know immediately if core will include the transaction in a future ledger. Detailed information about this can be found here: https://developers.stellar.org/api/errors/http-status-codes/horizon-specific/timeout/

We're looking at improving the docs to make all of this clearer. I hope this helps.

bartekn added the horizon label Feb 12, 2018

MonsieurNicolas mentioned this issue Feb 13, 2018

transactions that fail during consensus are ignored by Horizon #309

Closed

bartekn mentioned this issue Feb 14, 2018

Horizon timeout after transaction is added to ledger stellar/quickstart#41

Closed

bartekn closed this as completed Feb 14, 2018

bartekn reopened this Jul 12, 2018

bartekn mentioned this issue Jul 16, 2018

Add information about timeouts in POST /transactions #541

Merged

bartekn closed this as completed in #541 Jul 16, 2018

bartekn mentioned this issue Jul 20, 2018

Error 504 When high TPS for long time #382

Closed

bartekn reopened this Aug 29, 2018

gituser mentioned this issue Sep 11, 2018

Submit transaction timeout but still complete stellar/stellar-core#1778

Closed

bartekn closed this as completed Dec 6, 2018

bartekn mentioned this issue Dec 7, 2018

Timeout while submitting transaction to stellar network #771

Closed

bartekn reopened this Jan 16, 2019

bartekn mentioned this issue Jan 16, 2019

Fix deadlock in txsub.System.Tick() and tx_bad_seq errors #815

Merged

bartekn closed this as completed in #815 Jan 17, 2019

gituser mentioned this issue Mar 25, 2019

504 Timeout From Horizon zulucrypto/stellar-api#42

Open

overcat mentioned this issue Dec 15, 2021

504 timeout error StellarCN/py-stellar-base#568

Closed

Horizon timeout after transaction is added to ledger #305

Horizon timeout after transaction is added to ledger #305

Comments

oryband commented Feb 12, 2018 • edited

oryband commented Feb 12, 2018 • edited

oryband commented Feb 13, 2018

oryband commented Feb 14, 2018

bartekn commented Feb 14, 2018

s-a-y commented Feb 26, 2018 • edited

andrenarchy commented Jul 9, 2018

andrenarchy commented Jul 9, 2018

bartekn commented Jul 12, 2018 • edited

bartekn commented Jul 12, 2018

andrenarchy commented Jul 12, 2018

bartekn commented Jul 16, 2018 • edited

andrenarchy commented Jul 17, 2018

bartekn commented Jul 17, 2018

andrenarchy commented Jul 17, 2018 • edited

s-a-y commented Aug 28, 2018

ramisra commented Aug 28, 2018 • edited

bartekn commented Aug 29, 2018

gituser commented Sep 11, 2018 • edited

i3bitcoin commented Sep 14, 2018

gituser commented Sep 14, 2018 • edited

bartekn commented Oct 19, 2018

gituser commented Dec 2, 2018 • edited

gituser commented Dec 2, 2018

slavkomae commented Dec 4, 2018 • edited

gituser commented Dec 4, 2018 • edited

bartekn commented Dec 6, 2018

oryband commented Dec 23, 2018

gituser commented Dec 28, 2018 • edited

gituser commented Dec 29, 2018

oryband commented Jan 17, 2019

bartekn commented Jan 17, 2019 • edited

oryband commented Jan 17, 2019

bartekn commented Jan 17, 2019

andrenarchy commented Jan 17, 2019

s-a-y commented Jan 17, 2019

oryband commented Jan 17, 2019

gituser commented Jan 18, 2019 • edited

TFiroozian commented Nov 6, 2019 • edited

ire-and-curses commented Nov 11, 2019

gituser commented Jan 7, 2021

ire-and-curses commented Jan 7, 2021

gituser commented Jan 8, 2021 • edited

ire-and-curses commented Jan 8, 2021

oryband commented Feb 12, 2018 •

edited

oryband commented Feb 12, 2018 •

edited

s-a-y commented Feb 26, 2018 •

edited

bartekn commented Jul 12, 2018 •

edited

bartekn commented Jul 16, 2018 •

edited

andrenarchy commented Jul 17, 2018 •

edited

ramisra commented Aug 28, 2018 •

edited

gituser commented Sep 11, 2018 •

edited

gituser commented Sep 14, 2018 •

edited

gituser commented Dec 2, 2018 •

edited

slavkomae commented Dec 4, 2018 •

edited

gituser commented Dec 4, 2018 •

edited

gituser commented Dec 28, 2018 •

edited

bartekn commented Jan 17, 2019 •

edited

gituser commented Jan 18, 2019 •

edited

TFiroozian commented Nov 6, 2019 •

edited

gituser commented Jan 8, 2021 •

edited