Skip to content
This repository was archived by the owner on Oct 20, 2023. It is now read-only.
This repository was archived by the owner on Oct 20, 2023. It is now read-only.

[nat]: nat records are freed up prematurely #17

Closed
Closed
[nat]: nat records are freed up prematurely#17
Assignees
legomushroom
Labels
bugSomething isn't workingnattcp
@legomushroom

Description

@legomushroom
legomushroom
opened on May 9, 2022

Recently used NAT records can be cleaned up as old. This causes long-open TCP sockets to be reset by kernel.

Reproduce steps

  1. Create a long-running TCP connection (~30-60 seconds).
  2. Send "data" messages around periodically.
  3. Notice that after some time the connection is closed by a RST message from kernel.

Expected behavior

Long-running TPC connections do not close unexpectedly.

Logs


2022-05-06T20:30:58.759556Z TRACE client:interface:nats:tcp:id: NAT handled (from downstream)
2022-05-06T20:30:58.759615Z TRACE client:interface:stream: to network: 

IP V4 (id: 0x7E33):
Source:................................127.0.0.1
Destination:.......................192.168.86.27
Checksum:.................................0x26CC
Flags:......................................0x02
Payload len:..................................32
DSCP:.......................................0x00
ECN:........................................0x00

TCP:
                                           
Ports:...............................58300 -> 3000
Flags [0x10]:................................[ACK]
Window:.....................................0x06D8
Payload len:.....................................0
SEQ:.......................0xB7A38826 (3080947750)
ACK:.......................0xFFFC47B7 (4294723511)
Expected ACK:..............0xB7A38827 (3080947751)
Options:
  No operation: []
  No operation: []
  Timestamps: 38584AC0 A8761BA6
Checksum:............................0x1BF8 (7160)
Data offset:...................................0x8
Urgent pointer:.............................0x0000


2022-05-06T20:30:58.759708Z TRACE client:interface:stream:sender: sending packet: 

IP V4 (id: 0x7E33):
Source:................................127.0.0.1
Destination:.......................192.168.86.27
Checksum:.................................0x26CC
Flags:......................................0x02
Payload len:..................................32
DSCP:.......................................0x00
ECN:........................................0x00

TCP:
                                           
Ports:...............................58300 -> 3000
Flags [0x10]:................................[ACK]
Window:.....................................0x06D8
Payload len:.....................................0
SEQ:.......................0xB7A38826 (3080947750)
ACK:.......................0xFFFC47B7 (4294723511)
Expected ACK:..............0xB7A38827 (3080947751)
Options:
  No operation: []
  No operation: []
  Timestamps: 38584AC0 A8761BA6
Checksum:............................0x1BF8 (7160)
Data offset:...................................0x8
Urgent pointer:.............................0x0000


2022-05-06T20:30:58.759834Z TRACE client:interface:nats:tcp:id: 🔙 NAT from upstream: not our packet
2022-05-06T20:30:58.765377Z TRACE client:interface:nats:tcp:id: overriding loopback source to original upstream IP address
2022-05-06T20:30:58.765426Z TRACE client:interface:nats:tcp:id: 🔙 NAT from upstream: not our packet
2022-05-06T20:30:58.765460Z TRACE client:interface:nats:tcp:id: overriding loopback source to original upstream IP address
2022-05-06T20:30:58.765482Z TRACE client:interface:nats:tcp:id: 🔙 NAT from upstream: not our packet
2022-05-06T20:30:58.765608Z TRACE client:interface:stream: in 

IP V4 (id: 0x7E34):
Source:...............................172.16.5.4
Destination:.......................192.168.86.27
Checksum:.................................0xF4B7
Flags:......................................0x02
Payload len:..................................32
DSCP:.......................................0x00
ECN:........................................0x00

TCP:
                                           
Ports:...............................53696 -> 3000
Flags [0x10]:................................[ACK]
Window:.....................................0x06D8
Payload len:.....................................0
SEQ:.......................0xB7A38826 (3080947750)
ACK:.......................0xFFFC5307 (4294726407)
Expected ACK:..............0xB7A38827 (3080947751)
Options:
  No operation: []
  No operation: []
  Timestamps: 38584AC4 A8761BAC
Checksum:...........................0xC7FE (51198)
Data offset:...................................0x8
Urgent pointer:.............................0x0000


2022-05-06T20:30:58.765706Z TRACE client:interface:nats:tcp:id: NAT handled (from downstream)
2022-05-06T20:30:58.765764Z TRACE client:interface:stream: to network: 

IP V4 (id: 0x7E34):
Source:................................127.0.0.1
Destination:.......................192.168.86.27
Checksum:.................................0x26CB
Flags:......................................0x02
Payload len:..................................32
DSCP:.......................................0x00
ECN:........................................0x00

TCP:
                                           
Ports:...............................58300 -> 3000
Flags [0x10]:................................[ACK]
Window:.....................................0x06D8
Payload len:.....................................0
SEQ:.......................0xB7A38826 (3080947750)
ACK:.......................0xFFFC5307 (4294726407)
Expected ACK:..............0xB7A38827 (3080947751)
Options:
  No operation: []
  No operation: []
  Timestamps: 38584AC4 A8761BAC
Checksum:............................0x109E (4254)
Data offset:...................................0x8
Urgent pointer:.............................0x0000


2022-05-06T20:30:58.765859Z TRACE client:interface:stream:sender: sending packet: 

IP V4 (id: 0x7E34):
Source:................................127.0.0.1
Destination:.......................192.168.86.27
Checksum:.................................0x26CB
Flags:......................................0x02
Payload len:..................................32
DSCP:.......................................0x00
ECN:........................................0x00

TCP:
                                           
Ports:...............................58300 -> 3000
Flags [0x10]:................................[ACK]
Window:.....................................0x06D8
Payload len:.....................................0
SEQ:.......................0xB7A38826 (3080947750)
ACK:.......................0xFFFC5307 (4294726407)
Expected ACK:..............0xB7A38827 (3080947751)
Options:
  No operation: []
  No operation: []
  Timestamps: 38584AC4 A8761BAC
Checksum:............................0x109E (4254)
Data offset:...................................0x8
Urgent pointer:.............................0x0000


2022-05-06T20:30:58.765990Z TRACE client:interface:stream: in 

IP V4 (id: 0x7E35):
Source:...............................172.16.5.4
Destination:.......................192.168.86.27
Checksum:.................................0xF4B6
Flags:......................................0x02
Payload len:..................................32
DSCP:.......................................0x00
ECN:........................................0x00

TCP:
                                           
Ports:...............................53696 -> 3000
Flags [0x10]:................................[ACK]
Window:.....................................0x06E1
Payload len:.....................................0
SEQ:.......................0xB7A38826 (3080947750)
ACK:.......................0xFFFC57B7 (4294727607)
Expected ACK:..............0xB7A38827 (3080947751)
Options:
  No operation: []
  No operation: []
  Timestamps: 38584AC5 A8761BAC
Checksum:...........................0xC7FE (51198)
Data offset:...................................0x8
Urgent pointer:.............................0x0000


2022-05-06T20:30:58.766001Z TRACE client:interface:nats:tcp:id: 🔙 NAT from upstream: not our packet
2022-05-06T20:30:58.847122Z TRACE client:interface:nats:tcp:id: NAT handled (from downstream)
2022-05-06T20:30:58.847210Z TRACE client:interface:stream: to network: 

IP V4 (id: 0x7E35):
Source:................................127.0.0.1
Destination:.......................192.168.86.27
Checksum:.................................0x26CA
Flags:......................................0x02
Payload len:..................................32
DSCP:.......................................0x00
ECN:........................................0x00

TCP:
                                           
Ports:...............................58300 -> 3000
Flags [0x10]:................................[ACK]
Window:.....................................0x06E1
Payload len:.....................................0
SEQ:.......................0xB7A38826 (3080947750)
ACK:.......................0xFFFC57B7 (4294727607)
Expected ACK:..............0xB7A38827 (3080947751)
Options:
  No operation: []
  No operation: []
  Timestamps: 38584AC5 A8761BAC
Checksum:............................0x0BE4 (3044)
Data offset:...................................0x8
Urgent pointer:.............................0x0000


2022-05-06T20:30:58.847307Z TRACE client:interface:stream:sender: sending packet: 

IP V4 (id: 0x7E35):
Source:................................127.0.0.1
Destination:.......................192.168.86.27
Checksum:.................................0x26CA
Flags:......................................0x02
Payload len:..................................32
DSCP:.......................................0x00
ECN:........................................0x00

TCP:
                                           
Ports:...............................58300 -> 3000
Flags [0x10]:................................[ACK]
Window:.....................................0x06E1
Payload len:.....................................0
SEQ:.......................0xB7A38826 (3080947750)
ACK:.......................0xFFFC57B7 (4294727607)
Expected ACK:..............0xB7A38827 (3080947751)
Options:
  No operation: []
  No operation: []
  Timestamps: 38584AC5 A8761BAC
Checksum:............................0x0BE4 (3044)
Data offset:...................................0x8
Urgent pointer:.............................0x0000


2022-05-06T20:30:58.847499Z TRACE client:interface:stream: in 

IP V4 (id: 0x7E36):
Source:...............................172.16.5.4
Destination:.......................192.168.86.27
Checksum:.................................0xF4B5
Flags:......................................0x02
Payload len:..................................32
DSCP:.......................................0x00
ECN:........................................0x00

TCP:
                                           
Ports:...............................53696 -> 3000
Flags [0x11]:............................[ACK FIN]
Window:.....................................0x06E1
Payload len:.....................................0
SEQ:.......................0xB7A38826 (3080947750)
ACK:.......................0xFFFC57B7 (4294727607)
Expected ACK:..............0xB7A38827 (3080947751)
Options:
  No operation: []
  No operation: []
  Timestamps: 38584AC6 A8761BAC
Checksum:...........................0xC7FE (51198)
Data offset:...................................0x8
Urgent pointer:.............................0x0000


2022-05-06T20:30:58.847618Z TRACE client:interface:nats:tcp:id: NAT handled (from upstream)
2022-05-06T20:30:58.847686Z DEBUG client:interface:network: out 

IP V4 (id: 0x00):
Source:............................192.168.86.27
Destination:..........................172.16.5.4
Checksum:.................................0x6E3C
Flags:......................................0x02
Payload len:................................1232
DSCP:.......................................0x00
ECN:........................................0x00

TCP:
                                           
Ports:...............................3000 -> 53696
Flags [0x18]:............................[ACK PSH]
Window:.....................................0x0808
Payload len:..................................1200
SEQ:.......................0xFFFC5307 (4294726407)
ACK:.......................0xB7A38826 (3080947750)
Expected ACK:..............0xFFFC57B7 (4294727607)
Options:
  No operation: []
  No operation: []
  Timestamps: A8761C1D 38584AC4
Checksum:...........................0x3A7E (14974)
Data offset:...................................0x8
Urgent pointer:.............................0x0000


2022-05-06T20:30:58.847775Z TRACE client:interface:nats:tcp:id: NAT handled (from downstream)
2022-05-06T20:30:58.847830Z TRACE client:interface:stream: to network: 

IP V4 (id: 0x7E36):
Source:................................127.0.0.1
Destination:.......................192.168.86.27
Checksum:.................................0x26C9
Flags:......................................0x02
Payload len:..................................32
DSCP:.......................................0x00
ECN:........................................0x00

TCP:
                                           
Ports:...............................58300 -> 3000
Flags [0x11]:............................[ACK FIN]
Window:.....................................0x06E1
Payload len:.....................................0
SEQ:.......................0xB7A38826 (3080947750)
ACK:.......................0xFFFC57B7 (4294727607)
Expected ACK:..............0xB7A38827 (3080947751)
Options:
  No operation: []
  No operation: []
  Timestamps: 38584AC6 A8761BAC
Checksum:............................0x0BE2 (3042)
Data offset:...................................0x8
Urgent pointer:.............................0x0000


2022-05-06T20:30:58.847917Z TRACE client:interface:stream:sender: sending packet: 

IP V4 (id: 0x7E36):
Source:................................127.0.0.1
Destination:.......................192.168.86.27
Checksum:.................................0x26C9
Flags:......................................0x02
Payload len:..................................32
DSCP:.......................................0x00
ECN:........................................0x00

TCP:
                                           
Ports:...............................58300 -> 3000
Flags [0x11]:............................[ACK FIN]
Window:.....................................0x06E1
Payload len:.....................................0
SEQ:.......................0xB7A38826 (3080947750)
ACK:.......................0xFFFC57B7 (4294727607)
Expected ACK:..............0xB7A38827 (3080947751)
Options:
  No operation: []
  No operation: []
  Timestamps: 38584AC6 A8761BAC
Checksum:............................0x0BE2 (3042)
Data offset:...................................0x8
Urgent pointer:.............................0x0000


2022-05-06T20:30:58.848014Z TRACE client:interface:nats:tcp:id: 🔙 NAT from upstream: not our packet
2022-05-06T20:30:58.848047Z TRACE client:interface:nats:tcp:id: 🔙 NAT from upstream: not our packet
2022-05-06T20:30:58.848078Z TRACE client:interface:nats:tcp:id: NAT handled (from upstream)
2022-05-06T20:30:58.848129Z DEBUG client:interface:network: out 

IP V4 (id: 0x00):
Source:............................192.168.86.27
Destination:..........................172.16.5.4
Checksum:.................................0x72EC
Flags:......................................0x02
Payload len:..................................32
DSCP:.......................................0x00
ECN:........................................0x00

TCP:
                                           
Ports:...............................3000 -> 53696
Flags [0x10]:................................[ACK]
Window:.....................................0x0808
Payload len:.....................................0
SEQ:.......................0xFFFC57B7 (4294727607)
ACK:.......................0xB7A38827 (3080947751)
Expected ACK:..............0xFFFC57B8 (4294727608)
Options:
  No operation: []
  No operation: []
  Timestamps: A8761C1D 38584AC6
Checksum:...........................0xEA32 (59954)
Data offset:...................................0x8
Urgent pointer:.............................0x0000


2022-05-06T20:30:58.849922Z TRACE client:interface:nats:tcp:id: NAT handled (from upstream)
2022-05-06T20:30:58.850017Z DEBUG client:interface:network: out 

IP V4 (id: 0x00):
Source:............................192.168.86.27
Destination:..........................172.16.5.4
Checksum:.................................0x72EC
Flags:......................................0x02
Payload len:..................................32
DSCP:.......................................0x00
ECN:........................................0x00

TCP:
                                           
Ports:...............................3000 -> 53696
Flags [0x11]:............................[ACK FIN]
Window:.....................................0x0808
Payload len:.....................................0
SEQ:.......................0xFFFC57B7 (4294727607)
ACK:.......................0xB7A38827 (3080947751)
Expected ACK:..............0xFFFC57B8 (4294727608)
Options:
  No operation: []
  No operation: []
  Timestamps: A8761C1F 38584AC6
Checksum:...........................0xEA2F (59951)
Data offset:...................................0x8
Urgent pointer:.............................0x0000


2022-05-06T20:30:58.852925Z TRACE client:interface:nats:tcp:id: overriding loopback source to original upstream IP address
2022-05-06T20:30:58.852969Z TRACE client:interface:nats:tcp:id: 🔙 NAT from upstream: not our packet
2022-05-06T20:30:58.852997Z TRACE client:interface:nats:tcp:id: overriding loopback source to original upstream IP address
2022-05-06T20:30:58.853017Z TRACE client:interface:nats:tcp:id: 🔙 NAT from upstream: not our packet
2022-05-06T20:30:58.875327Z TRACE client:interface:nats:tcp:id: overriding loopback source to original upstream IP address
2022-05-06T20:30:58.875378Z TRACE client:interface:nats:tcp:id: 🔙 NAT from upstream: not our packet
2022-05-06T20:30:58.875407Z TRACE client:interface:nats:tcp:id: overriding loopback source to original upstream IP address
2022-05-06T20:30:58.875428Z TRACE client:interface:nats:tcp:id: 🔙 NAT from upstream: not our packet
2022-05-06T20:30:58.875543Z TRACE client:interface:stream: in 

IP V4 (id: 0x7E37):
Source:...............................172.16.5.4
Destination:.......................192.168.86.27
Checksum:.................................0xF4A8
Flags:......................................0x02
Payload len:..................................44
DSCP:.......................................0x00
ECN:........................................0x00

TCP:
                                           
Ports:...............................53696 -> 3000
Flags [0x10]:................................[ACK]
Window:.....................................0x06E1
Payload len:.....................................0
SEQ:.......................0xB7A38827 (3080947751)
ACK:.......................0xFFFC57B7 (4294727607)
Expected ACK:..............0xB7A38828 (3080947752)
Options:
  No operation: []
  No operation: []
  Timestamps: 38584B33 A8761C1D
  No operation: []
  No operation: []
  SACK: [FF, FC, 53, 07, FF, FC, 57, B7]
Checksum:...........................0xC80A (51210)
Data offset:...................................0xB
Urgent pointer:.............................0x0000


2022-05-06T20:30:58.875635Z TRACE client:interface:nats:tcp:id: NAT handled (from downstream)
2022-05-06T20:30:58.875661Z TRACE client:interface:nats: removed NAT [tcp_172.16.5.4:192.168.86.27_53696:3000]
2022-05-06T20:30:58.875704Z TRACE client:interface:nats:tcp:firewall: dropping
2022-05-06T20:30:58.875723Z TRACE client:interface:nats:tcp:firewall: removing TCP drop rule
2022-05-06T20:30:58.876356Z TRACE client:interface:nats:tcp:firewall: rule removed
2022-05-06T20:30:58.876392Z TRACE client:interface:nats:tcp:firewall: dropped
2022-05-06T20:30:58.876482Z TRACE client:interface:stream: to network: 

IP V4 (id: 0x7E37):
Source:................................127.0.0.1
Destination:.......................192.168.86.27
Checksum:.................................0x26BC
Flags:......................................0x02
Payload len:..................................44
DSCP:.......................................0x00
ECN:........................................0x00

TCP:
                                           
Ports:...............................58300 -> 3000
Flags [0x10]:................................[ACK]
Window:.....................................0x06E1
Payload len:.....................................0
SEQ:.......................0xB7A38827 (3080947751)
ACK:.......................0xFFFC57B7 (4294727607)
Expected ACK:..............0xB7A38828 (3080947752)
Options:
  No operation: []
  No operation: []
  Timestamps: 38584B33 A8761C1D
  No operation: []
  No operation: []
  SACK: [FF, FC, 53, 07, FF, FC, 57, B7]
Checksum:...........................0x2A34 (10804)
Data offset:...................................0xB
Urgent pointer:.............................0x0000


2022-05-06T20:30:58.876589Z TRACE client:interface:stream:sender: sending packet: 

IP V4 (id: 0x7E37):
Source:................................127.0.0.1
Destination:.......................192.168.86.27
Checksum:.................................0x26BC
Flags:......................................0x02
Payload len:..................................44
DSCP:.......................................0x00
ECN:........................................0x00

TCP:
                                           
Ports:...............................58300 -> 3000
Flags [0x10]:................................[ACK]
Window:.....................................0x06E1
Payload len:.....................................0
SEQ:.......................0xB7A38827 (3080947751)
ACK:.......................0xFFFC57B7 (4294727607)
Expected ACK:..............0xB7A38828 (3080947752)
Options:
  No operation: []
  No operation: []
  Timestamps: 38584B33 A8761C1D
  No operation: []
  No operation: []
  SACK: [FF, FC, 53, 07, FF, FC, 57, B7]
Checksum:...........................0x2A34 (10804)
Data offset:...................................0xB
Urgent pointer:.............................0x0000


2022-05-06T20:30:58.876723Z TRACE client:interface:stream: in 

IP V4 (id: 0x00):
Source:...............................172.16.5.4
Destination:.......................192.168.86.27
Checksum:.................................0x72EC
Flags:......................................0x02
Payload len:..................................32
DSCP:.......................................0x00
ECN:........................................0x00

TCP:
                                           
Ports:...............................53696 -> 3000
Flags [0x10]:................................[ACK]
Window:.....................................0x06E1
Payload len:.....................................0
SEQ:.......................0xB7A38827 (3080947751)
ACK:.......................0xFFFC57B8 (4294727608)
Expected ACK:..............0xB7A38828 (3080947752)
Options:
  No operation: []
  No operation: []
  Timestamps: 38584B34 A8761C1F
Checksum:...........................0xEAE8 (60136)
Data offset:...................................0x8
Urgent pointer:.............................0x0000

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingnattcp

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions