A misbehaving client can *crush* the scheduler

[michaelrsweet]

Version: 1.3.10
CUPS.org User: opher

Hello,
First, I didn't know if this qualifies as a security issue so I didn't mark it. Please feel free to mark it as such if it is.

A client (attached) running on:
Windows XP SP3 (up to date with security fixes)
Sun JDK 1.6.0_13

can crush or hang the CUPS server running on:
Fedora 10 i386 / cups 1:1.3.10-5.fc10
CentOS 5.2 x86_64 / cups 1.3.7-8

The Client is doing a Get-Jobs (like `lpq -a').
The server has 300 active jobs.
The client is broken: it thinks it got an incomplete response, exits abruptly causing the JVM/OS to send a TCP RST.
A .cap file from Wireshark (v1.0.7) running on the client is attached.

At that point CUPS scheduler either:
1. crashes
2. hangs
(rarely it takes a second try to kill the server)

the error_log shows this:
D [15/May/2009:18:06:49 +0300] cupsdAcceptClient: skipping getpeercon()
D [15/May/2009:18:06:49 +0300] cupsdAcceptClient: 1 from 10.236.33.36:631 (IPv4)
D [15/May/2009:18:06:49 +0300] cupsdReadClient: 1 POST / HTTP/1.0
D [15/May/2009:18:06:49 +0300] cupsdAuthorize: No authentication data provided.
D [15/May/2009:18:06:49 +0300] cupsdIsAuthorized: username=""
D [15/May/2009:18:06:49 +0300] Get-Jobs ipp://localhost/
D [15/May/2009:18:06:49 +0300] cupsdProcessIPPRequest: 1 status_code=0 (successful-ok)
D [15/May/2009:18:06:49 +0300] cupsdCloseClient: 1
D [15/May/2009:18:06:49 +0300] cupsdCloseClient: 7803248

To reproduce:

1. produce 300 active jobs on the CUPS server.
2. extract client.zip to any directory
3. execute: java -cp "cups-java-client-1.3.jar";. TestCupsGetJobs 10.236.33.136
   (replace 10.236.33.136 with your server address)

Note: I tried running the Client on a Linux machine but couldn't crash the CUPS server. I'm not sure but it seems that the JVM/Linux sent a TCP FIN (and not a TCP RST as on the MS-Windows machine).

Regards,
Opher.

[michaelrsweet]

CUPS.org User: opher

Note 2: It seems that when the server is running in LogLevel debug2 the client successfully reads the whole response. (So no abrupt exit).
So server needs to run in loglevel debug or info for it to crash.

[michaelrsweet]

CUPS.org User: mike

Please duplicate this against the current stable (1.3.10) release from cups.org (and not the patched up versions you get from Red Hat...) There are log messages that are not from the standard CUPS release.

If you can't reproduce with the cups.org release, we'll have to close this bug and you'll need to file a bug with Red Hat.

[michaelrsweet]

CUPS.org User: mike

We are unable to resolve this problem with the information provided. If you discover new information, please file a new STR referencing this one.

[michaelrsweet]

CUPS.org User: mike

Bringing this back from the dead since we finally have reproduction by Red Hat with a potential patch. From Jan Lieskovsky:

Hello vendors,

pretty long time ago Opher reported CUPS DoS (cupsd crash),
which can be performed from Win XP SP3 client - STR #3200:

[1] http://www.cups.org/str.php?L3200

We further investigated the issue (reproduced) and Tim Waugh
analyzed the reasons (see advisory) and provided patch
(attached cups-1.3.10-CVE-2009-3553.patch, cups-1.4.2-CVE-2009-3553.patch).

CVE identifier of CVE-2009-3553 has been already assigned
to this issue.

While the issue is meta-public, original ticket [1] suggests
to provide more information and reference [1] in new ticket,
so we would like to address this via coordinated CUPS release.

Embargo date has been set up to Wednesday, 2009-11-18,
12:00 UTC time. Please let us know, if this doesn't suit for
you for some reason.

Thanks && Regards, Jan.

Jan iankko Lieskovsky / Red Hat Security Response Team

CUPS background:

The Common UNIX® Printing System (CUPS) provides a portable
printing layer for UNIX operating systems.

CVE-2009-3553 flaw:

A pointer use-after-free flaw was found in the way CUPS
used to handle references in its abstract file descriptors
handling interface. Remote attacker could issue a specially-
-crafted get-printer-jobs request, leading to denial of
service (cupsd crash).

Flaw analysis (by Tim Waugh):

The CUPS scheduler uses an abstracted interface to handle select(),
poll(), epoll() and kqueue for systems that support them, in ascending
order of preference. The interface is simple, and the relevant parts
are:

• cupsdAddSelect, to add a particular file descriptor to the watch list,
  setting read/write callbacks and context
• cupsdRemoveSelect, to remove a particular file descriptor from the
  watch list
• cupsdDoSelect, to watch for changes and invoke callbacks as necessary

The cupsdDoSelect() function uses whichever system method is best to
watch for changes, and each time changes are discovered it loops through
the watch list. For each file descriptor there are changes on:

1. the watch list item has its reference count incremented in order to
   prevent problems if the read callback calls cupsdRemoveSelect for this
   file descriptor
2. if there is data readable, the read callback is invoked
3. if there is data writable, the write callback is invoked
4. the reference count is decremented again

Unfortunately this reference counting is not sufficient to prevent
problems if the read callback calls cupsdRemoveSelect, and the reason is
the context data that is set when cupsdAddSelect is called. Although
the data in the watch list is protected (including the pointer to the
context), the context data is not reference counted.

In main.c, cupsdDoSelect() is used to handle client connections and the
context is of type cupsd_client_t. When the read callback discovers
that the connection has been closed by the remote host, the file
descriptor is removed from the watch list using cupsdRemoveSelect, and
the cupsd_client_t object is freed.

If cupsdDoSelect() enters an iteration of its file descriptor loop
hoping to call both the read and write callbacks, but the read callback
closes the connection with cupsdCloseConnection, the write callback will
attempt to reference freed memory, leading to undefined behaviour.

A suggested fix for this is to avoid calling the write callback if
cupsdRemoveSelect() function was called from the read callback. I have
made a patch which does something similar by adding a 'changed' member
to the watch list structure, setting it in cupsdAddSelect() and
cupsdRemoveSelect(), clearing it before the read callback is called, and
checking it afterwards.

Affected CUPS versions: CUPS-1.3.* <= x <= CUPS-1.4.2 (latest)

CVE identifier: CVE-2009-3553 has been assigned to this issue

CRD date: Wednesday, 2009-11-18, 12:00 UTC time

PoC:

Pretty complex - unable to reproduce the issue from Linux client.
Requires Win XP SP3 client with Sun Java JDK-1.6.0-13 installed.

Java Cups Test Jobs client:

---

http://www.cups.org/strfiles/3200/Client.zip

Scenario:

---

Server has 300 active printing jobs.
Client issues Get-print-jobs request, thinks its incomplete,
aborts and resend TCP RST.

Impact:

---

CUPS-1.3.10 - clean crash (see attached debug && debug2 log).
CUPS-1.3.11 - crash (but logs aren't so "beauty").
CUPS-1.4.2 - no record for cupsd crash in /var/log/messages,
but cupsd just "is not running after exploitation."

See also "raw_steps.txt" and sample textonly.ppd for elaborated list
of steps.

Patches:

Attached are patches against 1.3.10 and 1.4.2 versions of CUPS,
provided by Tim Waugh.
diff -up cups-1.3.10/scheduler/select.c.str3200 cups-1.3.10/scheduler/select.c
--- cups-1.3.10/scheduler/select.c.str3200 2009-06-05 16:36:26.575877668 +0100
+++ cups-1.3.10/scheduler/select.c 2009-06-05 16:37:02.512752676 +0100
@@ -483,7 +483,7 @@ cupsdDoSelect(long timeout) /* I - Time
(*(fdptr->read_cb))(fdptr->data);
}

• if (fdptr->write_cb && event->filter == EVFILT_WRITE)

• if (fdptr->use > 1 && fdptr->write_cb && event->filter == EVFILT_WRITE)
  {
  cupsdLogMessage(CUPSD_LOG_DEBUG2, "cupsdDoSelect: Write on fd %d...",
  fdptr->fd);
  @@ -543,7 +543,7 @@ cupsdDoSelect(long timeout) /* I - Time
  (*(fdptr->read_cb))(fdptr->data);
  }

• if (fdptr->write_cb && (event->events & (EPOLLOUT | EPOLLERR | EPOLLHUP)))

• if (fdptr->use > 1 && fdptr->write_cb && (event->events & (EPOLLOUT | EPOLLERR | EPOLLHUP)))
  {
  cupsdLogMessage(CUPSD_LOG_DEBUG2, "cupsdDoSelect: Write on fd %d...",
  fdptr->fd);
  @@ -655,7 +655,7 @@ cupsdDoSelect(long timeout) /* I - Time
  (*(fdptr->read_cb))(fdptr->data);
  }

•  if (fdptr->write_cb && (pfd->revents & (POLLOUT | POLLERR | POLLHUP)))
  

•  if (fdptr->use > 1 && fdptr->write_cb && (pfd->revents & (POLLOUT | POLLERR | POLLHUP)))
  

  {
  cupsdLogMessage(CUPSD_LOG_DEBUG2, "cupsdDoSelect: Write on fd %d...",
  fdptr->fd);
  @@ -725,7 +725,7 @@ cupsdDoSelect(long timeout) /* I - Time
  (*(fdptr->read_cb))(fdptr->data);
  }

•  if (fdptr->write_cb && FD_ISSET(fdptr->fd, &cupsd_current_output))
  

•  if (fdptr->use > 1 && fdptr->write_cb && FD_ISSET(fdptr->fd, &cupsd_current_output))
  

  {
  cupsdLogMessage(CUPSD_LOG_DEBUG2, "cupsdDoSelect: Write on fd %d...",
  fdptr->fd);

diff -up cups-1.4.2/scheduler/select.c.str3200 cups-1.4.2/scheduler/select.c
--- cups-1.4.2/scheduler/select.c.str3200 2009-07-15 00:07:52.000000000 +0100
+++ cups-1.4.2/scheduler/select.c 2009-11-10 15:42:55.359665174 +0000
@@ -454,7 +454,7 @@ cupsdDoSelect(long timeout) /* I - Time
if (fdptr->read_cb && event->filter == EVFILT_READ)
(*(fdptr->read_cb))(fdptr->data);

• if (fdptr->write_cb && event->filter == EVFILT_WRITE)
• if (fdptr->use > 1 && fdptr->write_cb && event->filter == EVFILT_WRITE)
  (*(fdptr->write_cb))(fdptr->data);

release_fd(fdptr);
@@ -499,7 +499,7 @@ cupsdDoSelect(long timeout) /* I - Time
if (fdptr->read_cb && (event->events & (EPOLLIN | EPOLLERR | EPOLLHUP)))
(*(fdptr->read_cb))(fdptr->data);

• if (fdptr->write_cb && (event->events & (EPOLLOUT | EPOLLERR | EPOLLHUP)))
• if (fdptr->use > 1 && fdptr->write_cb && (event->events & (EPOLLOUT | EPOLLERR | EPOLLHUP)))
  (*(fdptr->write_cb))(fdptr->data);

release_fd(fdptr);
@@ -590,7 +590,7 @@ cupsdDoSelect(long timeout) /* I - Time
if (fdptr->read_cb && (pfd->revents & (POLLIN | POLLERR | POLLHUP)))
(*(fdptr->read_cb))(fdptr->data);

•  if (fdptr->write_cb && (pfd->revents & (POLLOUT | POLLERR | POLLHUP)))
  

•  if (fdptr->use > 1 && fdptr->write_cb && (pfd->revents & (POLLOUT | POLLERR | POLLHUP)))
  (*(fdptr->write_cb))(fdptr->data);
  

  release_fd(fdptr);
  @@ -645,7 +645,7 @@ cupsdDoSelect(long timeout) /* I - Time
  if (fdptr->read_cb && FD_ISSET(fdptr->fd, &cupsd_current_input))
  (*(fdptr->read_cb))(fdptr->data);

•  if (fdptr->write_cb && FD_ISSET(fdptr->fd, &cupsd_current_output))
  

•  if (fdptr->use > 1 && fdptr->write_cb && FD_ISSET(fdptr->fd, &cupsd_current_output))
  (*(fdptr->write_cb))(fdptr->data);
  

  release_fd(fdptr);

Prerequisites:

a, Linux host with running cupsd 'serversHostname', 'serversIP'
b, Windows client - 'clientsIP'
- was using "en_windows_xp_home_with_service_pack_3_x86_cd_x14-82413" and
- Java SE Development Kit 6u13 from:

"https://cds.sun.com/is-bin/INTERSHOP.enfinity/WFS/CDS-CDS_Developer-Site/en_US/-/USD/ViewProductDetail-Start?ProductRef=jdk-6u13-oth-JPR@CDS-CDS_Developer"

(Selected "Windows x64" platform).

CUPS Preparation phase:

a, remove all cups related stuff from the system

b, download and install cups

wget http://ftp.easysw.com/pub/cups/1.3.10/cups-1.3.10-source.tar.gz
tar xvzf cups-1.3.10-source.tar.gz
cd cups-1.3.10
./configure --prefix=/usr/local --enable-debug && make && make install

c, configure cups:

modify cups to listen on 'serversIP' and allow connection from 'clientsIP'

vi /usr/local/etc/cups/cupsd.conf

LogLevel info | debug | debug2

Only listen for connections from the local machine.

Listen localhost:631
Listen serversHostname:631
...

Restrict access to the server...

Allow From clientsIP Order allow,deny

add text-only printer to /usr/local/etc/cups/printers.conf

See attached textonly.ppd, you will need to change the serversHostname
in the:

DeviceURI ipp://serversHostname/printers/my_printer

row, other rows can be kept without change.

start cups and check if it recognizes 'test1' printer

/etc/init.d/cups start
/usr/local/bin/lpstat -v

schedule about 300 printing jobs

head -1000 /usr/share/dict/words > /tmp/file.txt
for i in seq 1 300; do /usr/local/bin/lp -d test1 /tmp/file.txt; done

JAVA client preparation phase:

Install the SUN JDK 1.6.0_13 Java on Windows

Set the JAVA_HOME variable

set JAVA_HOME=C:\Program Files\Java\jdk1.6.0_13\bin
(and potentially check via 'echo %JAVA_HOME%)

Download the malformed client and extract it

wget http://www.cups.org/strfiles/3200/Client.zip
unzip Client.zip && unzip cups-java-client-1.3.jar

Attack scenario:

Run the java client with 'serversIP' two times

java TestCupsGetJobs serversIP

(you should see something like:

"Requested X bytes; Got: Y bytes. Partial string:"
). Then repeat the request till you get:

"Connection refused:" (two times should be enough)

Now check /var/log/messages for
"*** glibc detected: /usr/local/sbin/cupsd: double free or corruption (top): address ***

You can also check /usr/local/var/log/cups/error_log for full
stack trace (in case 'debug2' log level was set up)

Printer configuration file for CUPS v1.3.10

Info Location DeviceURI ipp://serversHostname/printers/my_printer State Idle StateTime 1243606881 Accepting Yes Shared No JobSheets none none QuotaPeriod 0 PageLimit 0 KLimit 0 OpPolicy default ErrorPolicy stop-printer

[michaelrsweet]

CUPS.org User: mike

Fixed in Subversion repository.

[michaelrsweet]

"str3200.patch":

Index: scheduler/select.c

--- scheduler/select.c (revision 8887)
+++ scheduler/select.c (working copy)
@@ -454,7 +454,7 @@
if (fdptr->read_cb && event->filter == EVFILT_READ)
(*(fdptr->read_cb))(fdptr->data);

• if (fdptr->write_cb && event->filter == EVFILT_WRITE)
• if (fdptr->use > 1 && fdptr->write_cb && event->filter == EVFILT_WRITE)
  (*(fdptr->write_cb))(fdptr->data);

release_fd(fdptr);
@@ -499,7 +499,8 @@
if (fdptr->read_cb && (event->events & (EPOLLIN | EPOLLERR | EPOLLHUP)))
(*(fdptr->read_cb))(fdptr->data);

• if (fdptr->write_cb && (event->events & (EPOLLOUT | EPOLLERR | EPOLLHUP)))

• if (fdptr->use > 1 && fdptr->write_cb &&

•   (event->events & (EPOLLOUT | EPOLLERR | EPOLLHUP)))
  

  (*(fdptr->write_cb))(fdptr->data);

  release_fd(fdptr);
  @@ -590,7 +591,8 @@
  if (fdptr->read_cb && (pfd->revents & (POLLIN | POLLERR | POLLHUP)))
  (*(fdptr->read_cb))(fdptr->data);

•  if (fdptr->write_cb && (pfd->revents & (POLLOUT | POLLERR | POLLHUP)))
  

•  if (fdptr->use > 1 && fdptr->write_cb &&
  

•      (pfd->revents & (POLLOUT | POLLERR | POLLHUP)))
   (*(fdptr->write_cb))(fdptr->data);
  

  release_fd(fdptr);
  @@ -645,7 +647,8 @@
  if (fdptr->read_cb && FD_ISSET(fdptr->fd, &cupsd_current_input))
  (*(fdptr->read_cb))(fdptr->data);

•  if (fdptr->write_cb && FD_ISSET(fdptr->fd, &cupsd_current_output))
  

•  if (fdptr->use > 1 && fdptr->write_cb &&
  

•      FD_ISSET(fdptr->fd, &cupsd_current_output))
   (*(fdptr->write_cb))(fdptr->data);
  

  release_fd(fdptr);