drivers/net/telnet: Ignore unsupported subnegotiation data

[ankohuu]

Summary

Fix telnet sub-negotiation parsing so unsupported sub-option payload does not
leak into the first NSH command.

The issue happens when the telnet client sends IAC SB ... IAC SE
sub-negotiation data and the option is not handled by the current build,
for example when CONFIG_TELNET_SUPPORT_NAWS is disabled or
some new sub-options. In that case, the previous implementation could
return to normal input processing and let sub-negotiation payload
bytes appear as part of the first command read by NSH.

This change keeps consuming unsupported sub-negotiation payload until
IAC SE, and also handles escaped IAC IAC correctly inside
sub-negotiation payload so a literal 0xFF data byte is not mistaken for
the sub-negotiation terminator.

NAWS handling remains unchanged in functionality, but now follows the
same sub-negotiation framing rules more strictly.

Reference:

• Fixes BUG [[BUG] when connecting to NuttX over telnet an "&" character is inserted in the first command executed #18431]
• RFC 854 (telnet)
• RFC 1073 (NAWS)

Impact

bugfix

• Is new feature added? Is existing feature changed? NO
• Impact on user (will user need to adapt to change)? NO
• Impact on build (will build process change)? NO
• Impact on hardware (will arch(s) / board(s) / driver(s) change)? NO
• Impact on documentation (is update required / provided)? NO
• Impact on security (any sort of implications)? NO
• Impact on compatibility (backward/forward/interoperability)? NO
• Anything else to consider or add? None

Testing

I confirm that changes are verified on local setup and works as intended:

• Build Host(s): OS (Linux shun-tcubuntu 6.8.0-106-generic), CPU(Intel(R) Xeon(R) Platinum 8255C CPU), compiler(arm-gnu-toolchain-15.2.rel1-x86_64-aarch64-none-elf)
• Target(s): arch(aarch64), board:qemu-armv8a
• you can also provide steps on how to reproduce the problem and verify the change.
  • Launch Nuttx and run telnetd & from NSH as telnet server
  • Mock sub-negotiation client behavior using python code
  • Tested with and without CONFIG_TELNET_SUPPORT_NAWS config

Test python pseudo code

import socket, time

s = socket.create_connection(("host", port))

# IAC SB NAWS WIDTH[1] WIDTH[0] HEIGHT[1] HEIGHT[0] IAC SE
s.sendall(b"\xff\xfa\x1fwwhh\xff\xf0help\r\n")
wait
print(s.recv(8192).decode("latin1", errors="replace"))

# IAC SB TERMINAL-TYPE SEND IAC SE
s.sendall(b"\xff\xfa\x18xx\xff\xf0help\r\n")
wait
print(s.recv(8192).decode("latin1", errors="replace"))

# No sub-negotiation
s.sendall(b"help\r\n")
wait
print(s.recv(8192).decode("latin1", errors="replace"))

s.close()

Testing logs before change:

help command execution error
NuttShell (NSH) NuttX-12.12.0
nsh> nsh: wwhhhelp: command not found
nsh>

Testing logs after change with and without CONFIG_TELNET_SUPPORT_NAWS config:

ÿý when enable NAWS, should be response here.
NuttShell (NSH) NuttX-12.12.0
nsh> help usage:  help [-v] [<cmd>]

  .           cmp         free        mkrd        rmdir       uname
  [           dirname     get         mount       set         umount
  ?           df          help        mv          kill        unset
  alias       dmesg       hexdump     nfsmount    pkill       uptime
  unalias     echo        ifconfig    nslookup    sleep       watch
  arp         env         ifdown      pidof       usleep      wget
  basename    exec        ifup        printf      source      xd
  break       exit        ls          ps          test        wait
  cat         expr        md5         put         time
  cd          false       mkdir       pwd         true
  cp          fdinfo      mkfatfs     rm          truncate

Builtin Apps:
  dd            hello         nxplayer      ping          telnetd
  fdtdump       iperf         nxrecorder    renew
  getprime      nsh           ostest        sh
nsh>

[Copilot]

Pull request overview

Fixes a telnet parsing regression where subnegotiation (IAC SB …) bytes could bleed into the first NSH command when NAWS is disabled, by ensuring unsupported subnegotiation payload is consumed until IAC SE.

Changes:

• Adds a new parser state to discard unsupported telnet subnegotiation payload until the IAC SE terminator.
• Refactors the subnegotiation handling so NAWS (when enabled) continues to parse window size while other sub-options are ignored safely.
• Introduces explicit handling of the IAC SE terminator sequence in the state machine.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[linguini1]

Please fill out the PR template according to the contributing guide.

[acassis]

@ankohuu thank you for this fix! Please include the testing before and after this patch.

[cederom]

• Thank you @ankohuu :-)
• Please fill in the PR details using template (see https://github.com/apache/nuttx/blob/master/CONTRIBUTING.md).
• Please provide testing logs before and after the change.

[ankohuu]

Refactor to address copilot's two comments, already tested locally, will update pr details using template later.
@anchao Could you help to re-request copilot review? Seems I can't see that button.

Thank you all for your comments and help.

[anchao]

Refactor to address copilot's two comments, already tested locally, will update pr details using template later. @anchao Could you help to re-request copilot review? Seems I can't see that button.

Thank you all for your comments and help.

@ankohuu Got it. request the review from Copilot again. I think we can refer to Copilot's review comments, but we don't necessarily have to follow them strictly.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[cederom]

Just PR update and testing logs and we are ready to go, thanks @ankohuu :-)

Test log provided