Skip to content

Commit 790e4ab

Browse files
Julien-ElieJulien-Elie
committed
Initial implementation of the COMPRESS command in INN
Based on draft-murchison-nntp-compress-02 (still not published as an RFC). The COMPRESS command is an extension to the NNTP protocol to allow a connection to be effectively and efficiently compressed. News clients that also support that extension will be able to benefit from that bandwidth optimization and improvement in speed. Initial implementation in INN contributed by Julien Elie.
1 parent daf5070 commit 790e4ab

File tree

15 files changed

+467
-46
lines changed

15 files changed

+467
-46
lines changed

‎MANIFEST

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -622,6 +622,7 @@ nnrpd/sasl.c SASL authentication for nnrpd
622622
nnrpd/tls.c Transport layer security
623623
nnrpd/tls.h Transport layer security data types
624624
nnrpd/track.c Track client behavior
625+
nnrpd/zlib.c Compression support for nnrpd
625626
perl Perl libraries (Directory)
626627
perl/INN INN Perl modules (Directory)
627628
perl/INN/Config.pm.in INN::Config module

‎Makefile.global.in

Lines changed: 11 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -126,12 +126,19 @@ EXTLIB = @EXTLIB@
126126
LIBCC = $(LIBTOOLCC) $(CC)
127127
LIBLD = $(LIBTOOLLD) $(CC)
128128

129+
## zlib support. Additional flags and libraries used when compiling or
130+
## linking code that contains compression support.
131+
132+
ZLIB_CPPFLAGS = @ZLIB_CPPFLAGS@
133+
ZLIB_LDFLAGS = @ZLIB_LDFLAGS@
134+
ZLIB_LIBS = @ZLIB_LIBS@
135+
129136
## Berkeley DB support. If this support is configured, anything linking
130137
## against libstorage also needs to link against BDB_LDFLAGS and BDB_LIBS.
131-
132-
BDB_CPPFLAGS = @BDB_CPPFLAGS@ @ZLIB_CPPFLAGS@
133-
BDB_LDFLAGS = @BDB_LDFLAGS@ @ZLIB_LDFLAGS@
134-
BDB_LIBS = @BDB_LIBS@ @ZLIB_LIBS@
138+
139+
BDB_CPPFLAGS = @BDB_CPPFLAGS@ $(ZLIB_CPPFLAGS)
140+
BDB_LDFLAGS = @BDB_LDFLAGS@ $(ZLIB_LDFLAGS)
141+
BDB_LIBS = @BDB_LIBS@ $(ZLIB_LIBS)
135142

136143
## INN libraries. Nearly all INN programs are linked with libinn, and any
137144
## INN program that reads from or writes to article storage or overview is

‎configure.ac

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -324,11 +324,12 @@ if test x"$inn_enable_keywords" = x1 ; then
324324
fi
325325

326326
dnl Handle optional libraries and probing for their locations and component
327-
dnl libraries if needed. Support for zlib is handled later.
327+
dnl libraries if needed.
328328
INN_LIB_BDB_OPTIONAL
329329
INN_LIB_KRB5_OPTIONAL
330330
INN_LIB_OPENSSL_OPTIONAL
331331
INN_LIB_SASL_OPTIONAL
332+
INN_LIB_ZLIB_OPTIONAL
332333

333334
dnl If Kerberos is found, define KRB5_AUTH to auth_krb5 so as to build
334335
dnl that program. In case neither et/com_err.h nor kerberosv5/com_err.h
@@ -345,8 +346,8 @@ AS_IF([test x"$inn_use_KRB5" = xtrue],
345346
[AC_MSG_ERROR([cannot find usable com_err header])])])])])])
346347
AC_SUBST([KRB5_AUTH])
347348

348-
dnl If Berkeley DB is found, check the presence of its header.
349-
dnl Also, do not build with zlib support unless Berkeley DB is enabled.
349+
dnl If Berkeley DB is found, check the presence of its header and whether the
350+
dnl Berkeley DB library has ndbm support.
350351
AS_IF([test x"$inn_use_BDB" = xtrue],
351352
[inn_BDB_incroot=
352353
inn_BDB_header_found=
@@ -364,8 +365,7 @@ AS_IF([test x"$inn_use_BDB" = xtrue],
364365
[Define if you have the <db.h> header file.])],
365366
[inn_BDB_header_found=no])])
366367
AS_IF([test x"${inn_BDB_header_found}" = xyes],
367-
[INN_LIB_BDB_NDBM
368-
INN_LIB_ZLIB_OPTIONAL],
368+
[INN_LIB_BDB_NDBM],
369369
[AS_IF([test x"$with_bdb" = x],
370370
[BDB_CPPFLAGS=
371371
BDB_LDFLAGS=

‎doc/pod/install.pod

Lines changed: 14 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -413,15 +413,21 @@ explicitly passed to configure.
413413

414414
=item B<--with-zlib>=PATH
415415

416-
The ovdb storage method can optionally use compression. If B<--with-bdb>
417-
is set, and configure finds a suitable S<Berkeley DB> version, configure
418-
will by default also try to find the zlib library. INN will then be
419-
built with zlib support unless the B<--without-zlib> flag is explicitly
420-
passed to configure.
416+
Enables support for compression for news reading, which means a
417+
compression layer can be negotiated between your server and newsreaders
418+
supporting that NNTP extension.
419+
420+
Also enables support for compression with the ovdb storage method.
421421

422-
In case non-standard paths to the zlib library are used, one or both
423-
of the options B<--with-zlib-include> and B<--with-zlib-lib> can be
424-
given to configure with a path.
422+
This option requires that zlib be installed on your system (including the
423+
header files, not just the runtime libraries). If a path is given, it
424+
sets the installed directory of zlib. In case non-standard paths to the
425+
zlib library are used, one or both of the options B<--with-zlib-include>
426+
and B<--with-zlib-lib> can be given to configure with a path.
427+
428+
If the zlib library is found at configure time, INN will be built with
429+
compression support unless the B<--without-zlib> flag is explicitly
430+
passed to configure.
425431

426432
=item B<--with-openssl>=PATH
427433

‎doc/pod/news.pod

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,14 @@ true and UTC is really the local time zone of the server.
1313

1414
=item *
1515

16+
Julien Elie has implemented in B<nnrpd> the new COMPRESS command
17+
described in draft-murchison-nntp-compress that extends the NNTP protocol
18+
to allow a connection to be effectively and efficiently compressed.
19+
News clients that also support that extension will be able to benefit
20+
from that bandwidth optimization and improvement in speed.
21+
22+
=item *
23+
1624
When an encryption layer is negotiated during a successful use of the
1725
STARTTLS command, or after a successful authentication using a SASL
1826
mechanism which negotiates an encryption layer, B<nnrpd> now updates

‎include/inn/nntp.h

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,7 @@ enum nntp_code {
5555
NNTP_OK_BANNER_POST = 200,
5656
NNTP_OK_BANNER_NOPOST = 201,
5757
NNTP_OK_QUIT = 205,
58+
NNTP_OK_COMPRESS = 206,
5859
NNTP_OK_GROUP = 211,
5960
NNTP_OK_LIST = 215,
6061
NNTP_OK_ARTICLE = 220,

‎innd/nc.c

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,9 @@ static NCDISPATCH NCcommands[] = {
8888
READER command. */
8989
COMMAND_READER("ARTICLE"),
9090
COMMAND_READER("BODY"),
91+
#if defined(HAVE_ZLIB)
92+
COMMAND_READER("COMPRESS"),
93+
#endif /* HAVE_ZLIB */
9194
COMMAND_READER("DATE"),
9295
COMMAND_READER("GROUP"),
9396
COMMAND_READER("HDR"),

‎nnrpd/Makefile

Lines changed: 31 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -3,13 +3,13 @@
33
include ../Makefile.global
44

55
top = ..
6-
CFLAGS = $(GCFLAGS) $(SSL_CPPFLAGS) $(SASL_CPPFLAGS)
6+
CFLAGS = $(GCFLAGS) $(SSL_CPPFLAGS) $(SASL_CPPFLAGS) $(ZLIB_CPPFLAGS)
77

88
ALL = nnrpd
99

10-
SOURCES = article.c auth-ext.c cache.c group.c commands.c line.c \
10+
SOURCES = article.c auth-ext.c cache.c commands.c group.c line.c \
1111
list.c misc.c newnews.c nnrpd.c perl.c perm.c post.c \
12-
python.c sasl.c tls.c track.c
12+
python.c sasl.c tls.c track.c zlib.c
1313

1414
INCLUDES = cache.h nnrpd.h post.h tls.h
1515

@@ -34,10 +34,11 @@ clean clobber distclean maintclean:
3434

3535
NNRPDLIBS = $(LIBSTORAGE) $(LIBHIST) $(LIBINN) $(STORAGE_LIBS) \
3636
$(PERL_LIBS) $(PYTHON_LIBS) $(SSL_LDFLAGS) $(SSL_LIBS) \
37-
$(CRYPTO_LIBS) $(SASL_LDFLAGS) $(SASL_LIBS) $(LIBS)
37+
$(CRYPTO_LIBS) $(SASL_LDFLAGS) $(SASL_LIBS) \
38+
$(ZLIB_LDFLAGS) $(ZLIB_LIBS) $(LIBS)
3839

3940
.c.o:
40-
$(CC) $(CFLAGS) $(SSL_CPPFLAGS) $(SASL_CPPFLAGS) -c $<
41+
$(CC) $(CFLAGS) -c $<
4142

4243
perl.o: perl.c ; $(CC) $(CFLAGS) $(PERL_CPPFLAGS) -c perl.c
4344
python.o: python.c ; $(CC) $(CFLAGS) $(PYTHON_CPPFLAGS) -c python.c
@@ -104,19 +105,6 @@ cache.o: cache.c ../include/config.h ../include/inn/defines.h \
104105
../include/inn/tst.h ../include/inn/list.h ../include/inn/libinn.h \
105106
../include/inn/xmalloc.h ../include/inn/xwrite.h \
106107
../include/inn/storage.h ../include/inn/options.h cache.h
107-
group.o: group.c ../include/config.h ../include/inn/defines.h \
108-
../include/inn/system.h ../include/inn/macros.h \
109-
../include/inn/portable-macros.h ../include/inn/portable-stdbool.h \
110-
../include/inn/defines.h ../include/inn/options.h ../include/clibrary.h \
111-
../include/config.h ../include/inn/macros.h \
112-
../include/portable/stdbool.h ../include/inn/innconf.h nnrpd.h \
113-
../include/portable/macros.h ../include/portable/socket.h \
114-
../include/portable/getaddrinfo.h ../include/portable/getnameinfo.h \
115-
../include/inn/qio.h ../include/inn/libinn.h ../include/inn/xmalloc.h \
116-
../include/inn/xwrite.h ../include/inn/nntp.h ../include/inn/paths.h \
117-
../include/inn/storage.h ../include/inn/options.h \
118-
../include/inn/vector.h ../include/inn/timer.h ../include/inn/ov.h \
119-
../include/inn/history.h ../include/inn/storage.h
120108
commands.o: commands.c ../include/config.h ../include/inn/defines.h \
121109
../include/inn/system.h ../include/inn/macros.h \
122110
../include/inn/portable-macros.h ../include/inn/portable-stdbool.h \
@@ -132,6 +120,19 @@ commands.o: commands.c ../include/config.h ../include/inn/defines.h \
132120
../include/inn/ov.h ../include/inn/history.h ../include/inn/storage.h \
133121
../include/inn/innconf.h ../include/inn/messages.h \
134122
../include/inn/version.h tls.h
123+
group.o: group.c ../include/config.h ../include/inn/defines.h \
124+
../include/inn/system.h ../include/inn/macros.h \
125+
../include/inn/portable-macros.h ../include/inn/portable-stdbool.h \
126+
../include/inn/defines.h ../include/inn/options.h ../include/clibrary.h \
127+
../include/config.h ../include/inn/macros.h \
128+
../include/portable/stdbool.h ../include/inn/innconf.h nnrpd.h \
129+
../include/portable/macros.h ../include/portable/socket.h \
130+
../include/portable/getaddrinfo.h ../include/portable/getnameinfo.h \
131+
../include/inn/qio.h ../include/inn/libinn.h ../include/inn/xmalloc.h \
132+
../include/inn/xwrite.h ../include/inn/nntp.h ../include/inn/paths.h \
133+
../include/inn/storage.h ../include/inn/options.h \
134+
../include/inn/vector.h ../include/inn/timer.h ../include/inn/ov.h \
135+
../include/inn/history.h ../include/inn/storage.h
135136
line.o: line.c ../include/config.h ../include/inn/defines.h \
136137
../include/inn/system.h ../include/inn/macros.h \
137138
../include/inn/portable-macros.h ../include/inn/portable-stdbool.h \
@@ -290,3 +291,15 @@ track.o: track.c ../include/config.h ../include/inn/defines.h \
290291
../include/inn/xwrite.h ../include/inn/nntp.h ../include/inn/paths.h \
291292
../include/inn/storage.h ../include/inn/options.h \
292293
../include/inn/vector.h ../include/inn/timer.h
294+
zlib.o: zlib.c ../include/config.h ../include/inn/defines.h \
295+
../include/inn/system.h ../include/inn/macros.h \
296+
../include/inn/portable-macros.h ../include/inn/portable-stdbool.h \
297+
../include/inn/defines.h ../include/inn/options.h ../include/clibrary.h \
298+
../include/config.h ../include/inn/macros.h \
299+
../include/portable/stdbool.h ../include/inn/messages.h nnrpd.h \
300+
../include/portable/macros.h ../include/portable/socket.h \
301+
../include/portable/getaddrinfo.h ../include/portable/getnameinfo.h \
302+
../include/inn/qio.h ../include/inn/libinn.h ../include/inn/xmalloc.h \
303+
../include/inn/xwrite.h ../include/inn/nntp.h ../include/inn/paths.h \
304+
../include/inn/storage.h ../include/inn/options.h \
305+
../include/inn/vector.h ../include/inn/timer.h

‎nnrpd/article.c

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,23 @@ PushIOvHelper(struct iovec* vec, int* countp)
6464

6565
TMRstart(TMR_NNTPWRITE);
6666

67+
#if defined(HAVE_ZLIB)
68+
if (compression_layer_on) {
69+
int i;
70+
71+
for (i = 0; i < *countp; i++) {
72+
if (i+1 == *countp) {
73+
/* Time to flush the compressed output stream. */
74+
zstream_flush_needed = true;
75+
}
76+
write_buffer(vec[i].iov_base, vec[i].iov_len);
77+
}
78+
79+
*countp = 0;
80+
return;
81+
}
82+
#endif /* HAVE_ZLIB */
83+
6784
#ifdef HAVE_SASL
6885
if (sasl_conn && sasl_ssf) {
6986
int i;
@@ -196,6 +213,12 @@ static int highwater = 0;
196213
static void
197214
PushIOb(void)
198215
{
216+
#if defined(HAVE_ZLIB)
217+
/* Last line of a multi-line data block response.
218+
* Time to flush the compressed output stream. */
219+
zstream_flush_needed = true;
220+
#endif /* HAVE_ZLIB */
221+
199222
write_buffer(_IO_buffer_, highwater);
200223
highwater = 0;
201224
}

‎nnrpd/commands.c

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -264,6 +264,14 @@ CMDauthinfo(int ac, char *av[])
264264
char errorstr[BIG_BUFFER];
265265
int code;
266266

267+
#if defined(HAVE_ZLIB)
268+
/* If a compression layer is active, AUTHINFO is not possible. */
269+
if (compression_layer_on && !tls_compression_on) {
270+
Reply("%d Already using a compression layer\r\n", NNTP_ERR_ACCESS);
271+
return;
272+
}
273+
#endif
274+
267275
if (strcasecmp(av[1], "GENERIC") == 0) {
268276
char *logrec = Glom(av);
269277

‎nnrpd/line.c

Lines changed: 77 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@
2727
extern SSL *tls_conn;
2828
#endif
2929

30+
3031
/*
3132
** Free a previously allocated line structure.
3233
*/
@@ -86,6 +87,39 @@ line_doread(void *p, size_t len, int timeout UNUSED)
8687
ssize_t n;
8788

8889
do {
90+
#if defined(HAVE_ZLIB)
91+
/* Process data that may already be available in the zlib buffer. */
92+
if (compression_layer_on &&
93+
(zstream_in->avail_in > 0 || zstream_inflate_needed)) {
94+
int r;
95+
96+
zstream_in->next_out = p;
97+
zstream_in->avail_out = len;
98+
99+
r = inflate(zstream_in, Z_SYNC_FLUSH);
100+
101+
if (!(r == Z_OK || r == Z_BUF_ERROR || r == Z_STREAM_END)) {
102+
sysnotice("inflate() failed: %d; %s", r,
103+
zstream_in->msg != NULL ? zstream_in->msg :
104+
"no detail");
105+
n = -1;
106+
break;
107+
}
108+
109+
/* Check whether inflate() has finished to process its input.
110+
* If not, we need to call it again, even though avail_in is 0. */
111+
zstream_inflate_needed = (r != Z_STREAM_END);
112+
113+
if (zstream_in->avail_out < len) {
114+
/* Some data has been uncompressed. Treat it now. */
115+
n = len - zstream_in->avail_out;
116+
break;
117+
}
118+
/* If we reach here, then it means that inflate() needs more
119+
* input, so we go on reading data on the wire. */
120+
}
121+
#endif /* HAVE_ZLIB */
122+
89123
#ifdef HAVE_OPENSSL
90124
if (tls_conn) {
91125
int err;
@@ -129,7 +163,7 @@ line_doread(void *p, size_t len, int timeout UNUSED)
129163
const char *out;
130164
unsigned outlen;
131165
int r;
132-
166+
133167
if ((r = sasl_decode(sasl_conn, p, n, &out, &outlen)) == SASL_OK) {
134168
if (outlen > len) {
135169
sysnotice("sasl_decode() returned too much output");
@@ -150,6 +184,48 @@ line_doread(void *p, size_t len, int timeout UNUSED)
150184
}
151185
}
152186
#endif /* HAVE_SASL */
187+
188+
#if defined(HAVE_ZLIB)
189+
if (compression_layer_on && n > 0) {
190+
size_t zconsumed;
191+
192+
if (zstream_in->avail_in > 0 && zstream_in->next_in != Z_NULL) {
193+
zconsumed = zstream_in->next_in - zbuf_in;
194+
} else {
195+
zconsumed = 0;
196+
zbuf_in_allocated = 0;
197+
}
198+
199+
/* Transfer the data we have just read to zstream_in,
200+
* and loop to actually process it. */
201+
if ((ssize_t) (zbuf_in_size - zbuf_in_allocated) < n) {
202+
size_t newsize = zbuf_in_size * 2 + n;
203+
204+
/* Don't grow the buffer bigger than the maximum
205+
* article size we'll accept. */
206+
if (PERMaccessconf->localmaxartsize > NNTP_MAXLEN_COMMAND) {
207+
if (newsize > PERMaccessconf->localmaxartsize) {
208+
newsize = PERMaccessconf->localmaxartsize;
209+
}
210+
}
211+
if (newsize == zbuf_in_size) {
212+
warn("%s overflowed our zstream_in buffer (%lu)",
213+
Client.host, newsize);
214+
n = -1;
215+
break;
216+
}
217+
zbuf_in = xrealloc(zbuf_in, newsize);
218+
zbuf_in_size = newsize;
219+
}
220+
memcpy(zbuf_in + zbuf_in_allocated, p, n);
221+
zstream_in->next_in = zbuf_in + zconsumed;
222+
zstream_in->avail_in += n;
223+
zbuf_in_allocated += n;
224+
zstream_inflate_needed = true;
225+
/* Loop to actually inflate the compressed data we received. */
226+
n = 0;
227+
}
228+
#endif /* HAVE_ZLIB */
153229
} while (n == 0); /* Split SASL blob, need to read more data. */
154230

155231
return n;

0 commit comments

Comments
 (0)