Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] ERROR Error in _handle_one_transaction, closing #15217

Closed
iliasorazov opened this issue May 6, 2023 · 5 comments · Fixed by #15264
Closed

[Bug] ERROR Error in _handle_one_transaction, closing #15217

iliasorazov opened this issue May 6, 2023 · 5 comments · Fixed by #15264
Labels
1.7.0 bug Something isn't working CLI Farming full_node Linux Linux impact sync

Comments

@iliasorazov
Copy link

What happened?

If a node needs to sync it falls in a loop:

  1. it connects to a remote node to request blocks,
  2. it starts receiving transactions to mempool from the remote node and calls _handle_one_transaction
  3. there is no peak yet because syncing is in progress, it triggers assert in pre_validate_spendbundle,
  4. assert exception is caught and remote node is banned before it is able to send any block.

Relevant #14236 #14274

Version

1.7.0+

What platform are you using?

Linux

What ui mode are you using?

CLI

Relevant log output

Traceback (most recent call last):
File "/chia-blockchain/chia/full_node/full_node.py", in _handle_one_transaction
File "/chia-blockchain/chia/full_node/full_node.py", in add_transaction
  cost_result = await self.mempool_manager.pre_validate_spendbundle(...)
File "/chia-blockchain/chia/full_node/mempool_manager.py", in pre_validate_spendbundle
  assert self.peak is not None
AssertionError
@iliasorazov iliasorazov added the bug Something isn't working label May 6, 2023
@hbroer
Copy link

hbroer commented May 7, 2023 •
edited
Loading

same error here. Was afk 2 days and the farm stop working. I am on 1.8.0 too. Man normaly I stay away from new vewrsions because there are always problems. Just because you said "security problem"... Is there no QA?

My farm is ~70TB and it droped to ~10TB on spacepool. I just restarted but log fills up with this error and others instantly.

2023-05-07T18:09:38.614 full_node chia.full_node.full_node: ERROR    Error in _handle_one_transaction, closing: Traceback (most recent call last):
  File "[...]/chia/full_node/full_node.py", line 445, in _handle_one_transaction
    inc_status, err = await self.add_transaction(entry.transaction, entry.spend_name, peer, entry.test)
  File "[...]/chia/full_node/full_node.py", line 2221, in add_transaction
    cost_result = await self.mempool_manager.pre_validate_spendbundle(transaction, tx_bytes, spend_name)
  File "[...]/chia/full_node/mempool_manager.py", line 282, in pre_validate_spendbundle
    assert self.peak is not None
AssertionError

2023-05-07T18:09:38.675 full_node full_node_server        : ERROR    Exception: Error short batch syncing, could not fetch block at height 3628747, PeerInfo(_ip=IPv4Address('5.39.125.127'), _port=8444). Traceback (most recent call last):
  File "[...]/chia/server/ws_connection.py", line 393, in wrapped_coroutine
    result: Message = await coroutine
  File "[...]/chia/full_node/full_node_api.py", line 133, in new_peak
    await self.full_node.new_peak(request, peer)
  File "[...]/chia/full_node/full_node.py", line 737, in new_peak
    if await self.short_sync_batch(peer, uint32(max(curr_peak_height - 6, 0)), request.height):
  File "[...]/chia/full_node/full_node.py", line 557, in short_sync_batch
    raise ValueError(f"Error short batch syncing, could not fetch block at height {start_height}")
ValueError: Error short batch syncing, could not fetch block at height 3628747

2023-05-07T18:09:38.676 full_node full_node_server        : WARNING  Banning 5.39.125.127 for 10 seconds
2023-05-07T18:09:44.923 full_node chia.full_node.full_node: ERROR    Error in _handle_one_transaction, closing: Traceback (most recent call last):
  File "[...]/chia/full_node/full_node.py", line 445, in _handle_one_transaction
    inc_status, err = await self.add_transaction(entry.transaction, entry.spend_name, peer, entry.test)
  File "[...]/chia/full_node/full_node.py", line 2221, in add_transaction
    cost_result = await self.mempool_manager.pre_validate_spendbundle(transaction, tx_bytes, spend_name)
  File "[...]/chia/full_node/mempool_manager.py", line 282, in pre_validate_spendbundle
    assert self.peak is not None
AssertionError

2023-05-07T18:09:44.925 full_node chia.full_node.full_node: ERROR    Error in _handle_one_transaction, closing: Traceback (most recent call last):
  File "[...]/chia/full_node/full_node.py", line 445, in _handle_one_transaction
    inc_status, err = await self.add_transaction(entry.transaction, entry.spend_name, peer, entry.test)
  File "[...]/chia/full_node/full_node.py", line 2221, in add_transaction
    cost_result = await self.mempool_manager.pre_validate_spendbundle(transaction, tx_bytes, spend_name)
  File "[...]/chia/full_node/mempool_manager.py", line 282, in pre_validate_spendbundle
    assert self.peak is not None
AssertionError

2023-05-07T18:09:45.091 full_node full_node_server        : ERROR    Exception: Error short batch syncing, could not fetch block at height 3628747, PeerInfo(_ip=IPv4Address('23.126.61.27'), _port=8444). Traceback (most recent call last):
  File "[...]/chia/server/ws_connection.py", line 393, in wrapped_coroutine
    result: Message = await coroutine
  File "[...]/chia/full_node/full_node_api.py", line 133, in new_peak
    await self.full_node.new_peak(request, peer)
  File "[...]/chia/full_node/full_node.py", line 737, in new_peak
    if await self.short_sync_batch(peer, uint32(max(curr_peak_height - 6, 0)), request.height):
  File "[...]/chia/full_node/full_node.py", line 557, in short_sync_batch
    raise ValueError(f"Error short batch syncing, could not fetch block at height {start_height}")
ValueError: Error short batch syncing, could not fetch block at height 3628747

2023-05-07T18:09:45.093 full_node full_node_server        : WARNING  Banning 23.126.61.27 for 10 seconds
2023-05-07T18:09:48.360 full_node chia.full_node.full_node: ERROR    Error in _handle_one_transaction, closing: Traceback (most recent call last):
  File "[...]/chia/full_node/full_node.py", line 445, in _handle_one_transaction
    inc_status, err = await self.add_transaction(entry.transaction, entry.spend_name, peer, entry.test)
  File "[...]/chia/full_node/full_node.py", line 2221, in add_transaction
    cost_result = await self.mempool_manager.pre_validate_spendbundle(transaction, tx_bytes, spend_name)
  File "[...]/chia/full_node/mempool_manager.py", line 282, in pre_validate_spendbundle
    assert self.peak is not None
AssertionError

2023-05-07T18:09:48.361 full_node chia.full_node.full_node: ERROR    Error in _handle_one_transaction, closing: Traceback (most recent call last):
  File "[...]/chia/full_node/full_node.py", line 445, in _handle_one_transaction
    inc_status, err = await self.add_transaction(entry.transaction, entry.spend_name, peer, entry.test)
  File "[...]/chia/full_node/full_node.py", line 2221, in add_transaction
    cost_result = await self.mempool_manager.pre_validate_spendbundle(transaction, tx_bytes, spend_name)
  File "[...]/chia/full_node/mempool_manager.py", line 282, in pre_validate_spendbundle
    assert self.peak is not None
AssertionError

2023-05-07T18:09:48.363 full_node chia.full_node.full_node: ERROR    Error in _handle_one_transaction, closing: Traceback (most recent call last):
  File "[...]/chia/full_node/full_node.py", line 445, in _handle_one_transaction
    inc_status, err = await self.add_transaction(entry.transaction, entry.spend_name, peer, entry.test)
  File "[...]/chia/full_node/full_node.py", line 2221, in add_transaction
    cost_result = await self.mempool_manager.pre_validate_spendbundle(transaction, tx_bytes, spend_name)
  File "[...]/chia/full_node/mempool_manager.py", line 282, in pre_validate_spendbundle
    assert self.peak is not None
AssertionError

2023-05-07T18:09:48.364 full_node chia.full_node.full_node: ERROR    Error in _handle_one_transaction, closing: Traceback (most recent call last):
  File "[...]/chia/full_node/full_node.py", line 445, in _handle_one_transaction
    inc_status, err = await self.add_transaction(entry.transaction, entry.spend_name, peer, entry.test)
  File "[...]/chia/full_node/full_node.py", line 2221, in add_transaction
    cost_result = await self.mempool_manager.pre_validate_spendbundle(transaction, tx_bytes, spend_name)
  File "[...]/chia/full_node/mempool_manager.py", line 282, in pre_validate_spendbundle
    assert self.peak is not None
AssertionError

2023-05-07T18:09:48.902 full_node full_node_server        : ERROR    Exception: Error short batch syncing, could not fetch block at height 3628747, PeerInfo(_ip=IPv6Address('2a01:e0a:3d2:d940:38fe:9a0a:2b78:6d08'), _port=8444). Traceback (most recent call last):
  File "[...]/chia/server/ws_connection.py", line 393, in wrapped_coroutine
    result: Message = await coroutine
  File "[...]/chia/full_node/full_node_api.py", line 133, in new_peak
    await self.full_node.new_peak(request, peer)
  File "[...]/chia/full_node/full_node.py", line 737, in new_peak
    if await self.short_sync_batch(peer, uint32(max(curr_peak_height - 6, 0)), request.height):
  File "[...]/chia/full_node/full_node.py", line 557, in short_sync_batch
    raise ValueError(f"Error short batch syncing, could not fetch block at height {start_height}")
ValueError: Error short batch syncing, could not fetch block at height 3628747

2023-05-07T18:09:48.904 full_node full_node_server        : WARNING  Banning 2a01:e0a:3d2:d940:38fe:9a0a:2b78:6d08 for 10 seconds
2023-05-07T18:10:07.130 full_node chia.full_node.full_node: ERROR    Error in _handle_one_transaction, closing: Traceback (most recent call last):
  File "[...]/chia/full_node/full_node.py", line 445, in _handle_one_transaction
    inc_status, err = await self.add_transaction(entry.transaction, entry.spend_name, peer, entry.test)
  File "[...]/chia/full_node/full_node.py", line 2221, in add_transaction
    cost_result = await self.mempool_manager.pre_validate_spendbundle(transaction, tx_bytes, spend_name)
  File "[...]/chia/full_node/mempool_manager.py", line 282, in pre_validate_spendbundle
    assert self.peak is not None
AssertionError

2023-05-07T18:10:07.490 full_node full_node_server        : ERROR    Exception: Error short batch syncing, could not fetch block at height 3628747, PeerInfo(_ip=IPv4Address('175.161.124.201'), _port=8444). Traceback (most recent call last):
  File "[...]/chia/server/ws_connection.py", line 393, in wrapped_coroutine
    result: Message = await coroutine
  File "[...]/chia/full_node/full_node_api.py", line 133, in new_peak
    await self.full_node.new_peak(request, peer)
  File "[...]/chia/full_node/full_node.py", line 737, in new_peak
    if await self.short_sync_batch(peer, uint32(max(curr_peak_height - 6, 0)), request.height):
  File "[...]/chia/full_node/full_node.py", line 557, in short_sync_batch
    raise ValueError(f"Error short batch syncing, could not fetch block at height {start_height}")
ValueError: Error short batch syncing, could not fetch block at height 3628747

2023-05-07T18:10:07.491 full_node full_node_server        : WARNING  Banning 175.161.124.201 for 10 seconds
2023-05-07T18:10:28.963 full_node chia.full_node.full_node: ERROR    Error in _handle_one_transaction, closing: Traceback (most recent call last):
  File "[...]/chia/full_node/full_node.py", line 445, in _handle_one_transaction
    inc_status, err = await self.add_transaction(entry.transaction, entry.spend_name, peer, entry.test)
  File "[...]/chia/full_node/full_node.py", line 2221, in add_transaction
    cost_result = await self.mempool_manager.pre_validate_spendbundle(transaction, tx_bytes, spend_name)
  File "[...]/chia/full_node/mempool_manager.py", line 282, in pre_validate_spendbundle
    assert self.peak is not None
AssertionError

2023-05-07T18:10:28.964 full_node chia.full_node.full_node: ERROR    Error in _handle_one_transaction, closing: Traceback (most recent call last):
  File "[...]/chia/full_node/full_node.py", line 445, in _handle_one_transaction
    inc_status, err = await self.add_transaction(entry.transaction, entry.spend_name, peer, entry.test)
  File "[...]/chia/full_node/full_node.py", line 2221, in add_transaction
    cost_result = await self.mempool_manager.pre_validate_spendbundle(transaction, tx_bytes, spend_name)
  File "[...]/chia/full_node/mempool_manager.py", line 282, in pre_validate_spendbundle
    assert self.peak is not None
AssertionError

2023-05-07T18:10:28.966 full_node chia.full_node.full_node: ERROR    Error in _handle_one_transaction, closing: Traceback (most recent call last):
  File "[...]/chia/full_node/full_node.py", line 445, in _handle_one_transaction
    inc_status, err = await self.add_transaction(entry.transaction, entry.spend_name, peer, entry.test)
  File "[...]/chia/full_node/full_node.py", line 2221, in add_transaction
    cost_result = await self.mempool_manager.pre_validate_spendbundle(transaction, tx_bytes, spend_name)
  File "[...]/chia/full_node/mempool_manager.py", line 282, in pre_validate_spendbundle
    assert self.peak is not None
AssertionError

2023-05-07T18:10:28.967 full_node chia.full_node.full_node: ERROR    Error in _handle_one_transaction, closing: Traceback (most recent call last):
  File "[...]/chia/full_node/full_node.py", line 445, in _handle_one_transaction
    inc_status, err = await self.add_transaction(entry.transaction, entry.spend_name, peer, entry.test)
  File "[...]/chia/full_node/full_node.py", line 2221, in add_transaction
    cost_result = await self.mempool_manager.pre_validate_spendbundle(transaction, tx_bytes, spend_name)
  File "[...]/chia/full_node/mempool_manager.py", line 282, in pre_validate_spendbundle
    assert self.peak is not None
AssertionError

2023-05-07T18:10:28.969 full_node chia.full_node.full_node: ERROR    Error in _handle_one_transaction, closing: Traceback (most recent call last):
  File "[...]/chia/full_node/full_node.py", line 445, in _handle_one_transaction
    inc_status, err = await self.add_transaction(entry.transaction, entry.spend_name, peer, entry.test)
  File "[...]/chia/full_node/full_node.py", line 2221, in add_transaction
    cost_result = await self.mempool_manager.pre_validate_spendbundle(transaction, tx_bytes, spend_name)
  File "[...]/chia/full_node/mempool_manager.py", line 282, in pre_validate_spendbundle
    assert self.peak is not None
AssertionError

2023-05-07T18:10:28.970 full_node chia.full_node.full_node: ERROR    Error in _handle_one_transaction, closing: Traceback (most recent call last):
  File "[...]/chia/full_node/full_node.py", line 445, in _handle_one_transaction
    inc_status, err = await self.add_transaction(entry.transaction, entry.spend_name, peer, entry.test)
  File "[...]/chia/full_node/full_node.py", line 2221, in add_transaction
    cost_result = await self.mempool_manager.pre_validate_spendbundle(transaction, tx_bytes, spend_name)
  File "[...]/chia/full_node/mempool_manager.py", line 282, in pre_validate_spendbundle
    assert self.peak is not None
AssertionError

2023-05-07T18:10:28.972 full_node chia.full_node.full_node: ERROR    Error in _handle_one_transaction, closing: Traceback (most recent call last):
  File "[...]/chia/full_node/full_node.py", line 445, in _handle_one_transaction
    inc_status, err = await self.add_transaction(entry.transaction, entry.spend_name, peer, entry.test)
  File "[...]/chia/full_node/full_node.py", line 2221, in add_transaction
    cost_result = await self.mempool_manager.pre_validate_spendbundle(transaction, tx_bytes, spend_name)
  File "[...]/chia/full_node/mempool_manager.py", line 282, in pre_validate_spendbundle
    assert self.peak is not None
AssertionError

2023-05-07T18:10:28.973 full_node chia.full_node.full_node: ERROR    Error in _handle_one_transaction, closing: Traceback (most recent call last):
  File "[...]/chia/full_node/full_node.py", line 445, in _handle_one_transaction
    inc_status, err = await self.add_transaction(entry.transaction, entry.spend_name, peer, entry.test)
  File "[...]/chia/full_node/full_node.py", line 2221, in add_transaction
    cost_result = await self.mempool_manager.pre_validate_spendbundle(transaction, tx_bytes, spend_name)
  File "[...]/chia/full_node/mempool_manager.py", line 282, in pre_validate_spendbundle
    assert self.peak is not None
AssertionError

2023-05-07T18:10:28.975 full_node chia.full_node.full_node: ERROR    Error in _handle_one_transaction, closing: Traceback (most recent call last):
  File "[...]/chia/full_node/full_node.py", line 445, in _handle_one_transaction
    inc_status, err = await self.add_transaction(entry.transaction, entry.spend_name, peer, entry.test)
  File "[...]/chia/full_node/full_node.py", line 2221, in add_transaction
    cost_result = await self.mempool_manager.pre_validate_spendbundle(transaction, tx_bytes, spend_name)
  File "[...]/chia/full_node/mempool_manager.py", line 282, in pre_validate_spendbundle
    assert self.peak is not None
AssertionError

2023-05-07T18:10:28.976 full_node chia.full_node.full_node: ERROR    Error in _handle_one_transaction, closing: Traceback (most recent call last):
  File "[...]/chia/full_node/full_node.py", line 445, in _handle_one_transaction
    inc_status, err = await self.add_transaction(entry.transaction, entry.spend_name, peer, entry.test)
  File "[...]/chia/full_node/full_node.py", line 2221, in add_transaction
    cost_result = await self.mempool_manager.pre_validate_spendbundle(transaction, tx_bytes, spend_name)
  File "[...]/chia/full_node/mempool_manager.py", line 282, in pre_validate_spendbundle
    assert self.peak is not None
AssertionError

2023-05-07T18:10:29.278 full_node chia.full_node.full_node: ERROR    Error in _handle_one_transaction, closing: Traceback (most recent call last):
  File "[...]/chia/full_node/full_node.py", line 445, in _handle_one_transaction
    inc_status, err = await self.add_transaction(entry.transaction, entry.spend_name, peer, entry.test)
  File "[...]/chia/full_node/full_node.py", line 2221, in add_transaction
    cost_result = await self.mempool_manager.pre_validate_spendbundle(transaction, tx_bytes, spend_name)
  File "[...]/chia/full_node/mempool_manager.py", line 282, in pre_validate_spendbundle
    assert self.peak is not None
AssertionError

2023-05-07T18:10:29.280 full_node chia.full_node.full_node: ERROR    Error in _handle_one_transaction, closing: Traceback (most recent call last):
  File "[...]/chia/full_node/full_node.py", line 445, in _handle_one_transaction
    inc_status, err = await self.add_transaction(entry.transaction, entry.spend_name, peer, entry.test)
  File "[...]/chia/full_node/full_node.py", line 2221, in add_transaction
    cost_result = await self.mempool_manager.pre_validate_spendbundle(transaction, tx_bytes, spend_name)
  File "[...]/chia/full_node/mempool_manager.py", line 282, in pre_validate_spendbundle
    assert self.peak is not None
AssertionError

2023-05-07T18:10:29.281 full_node chia.full_node.full_node: ERROR    Error in _handle_one_transaction, closing: Traceback (most recent call last):
  File "[...]/chia/full_node/full_node.py", line 445, in _handle_one_transaction
    inc_status, err = await self.add_transaction(entry.transaction, entry.spend_name, peer, entry.test)
  File "[...]/chia/full_node/full_node.py", line 2221, in add_transaction
    cost_result = await self.mempool_manager.pre_validate_spendbundle(transaction, tx_bytes, spend_name)
  File "[...]/chia/full_node/mempool_manager.py", line 282, in pre_validate_spendbundle
    assert self.peak is not None
AssertionError

2023-05-07T18:10:29.283 full_node chia.full_node.full_node: ERROR    Error in _handle_one_transaction, closing: Traceback (most recent call last):
  File "[...]/chia/full_node/full_node.py", line 445, in _handle_one_transaction
    inc_status, err = await self.add_transaction(entry.transaction, entry.spend_name, peer, entry.test)
  File "[...]/chia/full_node/full_node.py", line 2221, in add_transaction
    cost_result = await self.mempool_manager.pre_validate_spendbundle(transaction, tx_bytes, spend_name)
  File "[...]/chia/full_node/mempool_manager.py", line 282, in pre_validate_spendbundle
    assert self.peak is not None
AssertionError

2023-05-07T18:10:29.284 full_node chia.full_node.full_node: ERROR    Error in _handle_one_transaction, closing: Traceback (most recent call last):
  File "[...]/chia/full_node/full_node.py", line 445, in _handle_one_transaction
    inc_status, err = await self.add_transaction(entry.transaction, entry.spend_name, peer, entry.test)
  File "[...]/chia/full_node/full_node.py", line 2221, in add_transaction
    cost_result = await self.mempool_manager.pre_validate_spendbundle(transaction, tx_bytes, spend_name)
  File "[...]/chia/full_node/mempool_manager.py", line 282, in pre_validate_spendbundle
    assert self.peak is not None
AssertionError

2023-05-07T18:10:29.286 full_node chia.full_node.full_node: ERROR    Error in _handle_one_transaction, closing: Traceback (most recent call last):
  File "[...]/chia/full_node/full_node.py", line 445, in _handle_one_transaction
    inc_status, err = await self.add_transaction(entry.transaction, entry.spend_name, peer, entry.test)
  File "[...]/chia/full_node/full_node.py", line 2221, in add_transaction
    cost_result = await self.mempool_manager.pre_validate_spendbundle(transaction, tx_bytes, spend_name)
  File "[...]/chia/full_node/mempool_manager.py", line 282, in pre_validate_spendbundle
    assert self.peak is not None
AssertionError

2023-05-07T18:10:29.476 full_node full_node_server        : ERROR    Exception: Error short batch syncing, could not fetch block at height 3628747, PeerInfo(_ip=IPv4Address('62.176.17.72'), _port=8444). Traceback (most recent call last):
  File "[...]/chia/server/ws_connection.py", line 393, in wrapped_coroutine
    result: Message = await coroutine
  File "[...]/chia/full_node/full_node_api.py", line 133, in new_peak
    await self.full_node.new_peak(request, peer)
  File "[...]/chia/full_node/full_node.py", line 737, in new_peak
    if await self.short_sync_batch(peer, uint32(max(curr_peak_height - 6, 0)), request.height):
  File "[...]/chia/full_node/full_node.py", line 557, in short_sync_batch
    raise ValueError(f"Error short batch syncing, could not fetch block at height {start_height}")
ValueError: Error short batch syncing, could not fetch block at height 3628747

2023-05-07T18:10:29.478 full_node full_node_server        : WARNING  Banning 62.176.17.72 for 10 seconds
2023-05-07T18:10:29.613 full_node full_node_server        : ERROR    Exception: Error short batch syncing, could not fetch block at height 3628747, PeerInfo(_ip=IPv4Address('88.97.26.58'), _port=8444). Traceback (most recent call last):
  File "[...]/chia/server/ws_connection.py", line 393, in wrapped_coroutine
    result: Message = await coroutine
  File "[...]/chia/full_node/full_node_api.py", line 133, in new_peak
    await self.full_node.new_peak(request, peer)
  File "[...]/chia/full_node/full_node.py", line 737, in new_peak
    if await self.short_sync_batch(peer, uint32(max(curr_peak_height - 6, 0)), request.height):
  File "[...]/chia/full_node/full_node.py", line 557, in short_sync_batch
    raise ValueError(f"Error short batch syncing, could not fetch block at height {start_height}")
ValueError: Error short batch syncing, could not fetch block at height 3628747

2023-05-07T18:10:29.615 full_node full_node_server        : WARNING  Banning 88.97.26.58 for 10 seconds
2023-05-07T18:10:44.462 full_node full_node_server        : WARNING  Connection failed to wss://[2600:1f13:c55:4403:67e2:2fea:fc65:cdff]:8444/ws. Error: 403, message='Invalid response status', url=URL('wss://[2600:1f13:c55:4403:67e2:2fea:fc65:cdff]:8444/ws')
2023-05-07T18:10:48.459 full_node chia.full_node.full_node: ERROR    Error in _handle_one_transaction, closing: Traceback (most recent call last):
  File "[...]/chia/full_node/full_node.py", line 445, in _handle_one_transaction
    inc_status, err = await self.add_transaction(entry.transaction, entry.spend_name, peer, entry.test)
  File "[...]/chia/full_node/full_node.py", line 2221, in add_transaction
    cost_result = await self.mempool_manager.pre_validate_spendbundle(transaction, tx_bytes, spend_name)
  File "[...]/chia/full_node/mempool_manager.py", line 282, in pre_validate_spendbundle
    assert self.peak is not None
AssertionError

2023-05-07T18:10:48.745 full_node full_node_server        : ERROR    Exception: Error short batch syncing, could not fetch block at height 3628747, PeerInfo(_ip=IPv4Address('93.116.1.208'), _port=8444). Traceback (most recent call last):
  File "[...]/chia/server/ws_connection.py", line 393, in wrapped_coroutine
    result: Message = await coroutine
  File "[...]/chia/full_node/full_node_api.py", line 133, in new_peak
    await self.full_node.new_peak(request, peer)
  File "[...]/chia/full_node/full_node.py", line 737, in new_peak
    if await self.short_sync_batch(peer, uint32(max(curr_peak_height - 6, 0)), request.height):
  File "[...]/chia/full_node/full_node.py", line 557, in short_sync_batch
    raise ValueError(f"Error short batch syncing, could not fetch block at height {start_height}")
ValueError: Error short batch syncing, could not fetch block at height 3628747

2023-05-07T18:10:48.751 full_node full_node_server        : WARNING  Banning 93.116.1.208 for 10 seconds
2023-05-07T18:10:56.629 full_node chia.full_node.full_node: ERROR    Error in _handle_one_transaction, closing: Traceback (most recent call last):
  File "[...]/chia/full_node/full_node.py", line 445, in _handle_one_transaction
    inc_status, err = await self.add_transaction(entry.transaction, entry.spend_name, peer, entry.test)
  File "[...]/chia/full_node/full_node.py", line 2221, in add_transaction
    cost_result = await self.mempool_manager.pre_validate_spendbundle(transaction, tx_bytes, spend_name)
  File "[...]/chia/full_node/mempool_manager.py", line 282, in pre_validate_spendbundle
    assert self.peak is not None
AssertionError

2023-05-07T18:10:56.630 full_node chia.full_node.full_node: ERROR    Error in _handle_one_transaction, closing: Traceback (most recent call last):
  File "[...]/chia/full_node/full_node.py", line 445, in _handle_one_transaction
    inc_status, err = await self.add_transaction(entry.transaction, entry.spend_name, peer, entry.test)
  File "[...]/chia/full_node/full_node.py", line 2221, in add_transaction
    cost_result = await self.mempool_manager.pre_validate_spendbundle(transaction, tx_bytes, spend_name)
  File "[...]/chia/full_node/mempool_manager.py", line 282, in pre_validate_spendbundle
    assert self.peak is not None
AssertionError

2023-05-07T18:10:56.632 full_node chia.full_node.full_node: ERROR    Error in _handle_one_transaction, closing: Traceback (most recent call last):
  File "[...]/chia/full_node/full_node.py", line 445, in _handle_one_transaction
    inc_status, err = await self.add_transaction(entry.transaction, entry.spend_name, peer, entry.test)
  File "[...]/chia/full_node/full_node.py", line 2221, in add_transaction
    cost_result = await self.mempool_manager.pre_validate_spendbundle(transaction, tx_bytes, spend_name)
  File "[...]/chia/full_node/mempool_manager.py", line 282, in pre_validate_spendbundle
    assert self.peak is not None
AssertionError

2023-05-07T18:10:56.740 full_node full_node_server        : ERROR    Exception: Error short batch syncing, could not fetch block at height 3628747, PeerInfo(_ip=IPv4Address('90.177.236.248'), _port=8444). Traceback (most recent call last):
  File "[...]/chia/server/ws_connection.py", line 393, in wrapped_coroutine
    result: Message = await coroutine
  File "[...]/chia/full_node/full_node_api.py", line 133, in new_peak
    await self.full_node.new_peak(request, peer)
  File "[...]/chia/full_node/full_node.py", line 737, in new_peak
    if await self.short_sync_batch(peer, uint32(max(curr_peak_height - 6, 0)), request.height):
  File "[...]/chia/full_node/full_node.py", line 557, in short_sync_batch
    raise ValueError(f"Error short batch syncing, could not fetch block at height {start_height}")
ValueError: Error short batch syncing, could not fetch block at height 3628747

@emlowe
Copy link
Contributor

emlowe commented May 11, 2023

This error shouldn't cause too many issues with the node as there isn't anything that can be done with this transaction until you get a peak. There is a try handler, so the node should continue to run. Generally, it's expected for a peer node to send us a peak before transactions but this is not guaranteed.

However, the worst part is that the peer is closed and banned, which could result in taking longer to get an actual peak as you have to find new peers. Leaving this issue open in order to better handle this case with improved logging and less banning of peers.

@iliasorazov
Copy link
Author

iliasorazov commented May 12, 2023 •
edited
Loading

Node is not crashing but it cant sync it bans one remote node after another eventually running out of options, dramatically slowing down the start time. Previous versions didn't ban node if it sends transaction before a block. It is a regression. Why did someone put an assert there and consider such situation an error? You have no peak, you have received a transaction, ok, ignore it silently and do not escalate it into an error.

@emlowe emlowe linked a pull request May 12, 2023 that will close this issue
don't add transactions to the mempool before it has a valid peak #15264
Merged
@emlowe
Copy link
Contributor

emlowe commented May 12, 2023

The 1.8.0 soft-fork required the height to validate spendbundles properly - it wasn't needed before at all, so there wasn't any need to flag the error case.

Thanks for the detailed bug report - we hope to have a fix soon

@emlowe
Copy link
Contributor

emlowe commented May 12, 2023

This should be fixed in 1.8.1 release

@emlowe emlowe closed this as completed May 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
1.7.0 bug Something isn't working CLI Farming full_node Linux Linux impact sync
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

don't add transactions to the mempool before it has a valid peak Chia-Network/chia-blockchain
4 participants
@hbroer @emlowe @iliasorazov @MumfMeisterT