Stream pkg INSERTs rather than caching everything.

Use libarchive (which we were linking against but not using) to handle
streaming the remote pkg_summary.  The copy of decompress.c is no longer
required.

We now stream one pkg_info record at a time, rather than loading the
entire contents in and then processing them all in one go.  They are
still processed as part of the same commit, so there is no performance
penalty (if anything this version is slightly faster).

Using the SmartOS 2015Q1 x86_64 repository as a test corpus, heap usage
for a clean "pkgin update" with this change goes down from 75MB to 29MB.

Makefile.in

@@ -8,8 +8,7 @@ SRCS=		main.c summary.c tools.c pkgindb.c depends.c actions.c \
 		pkgindb_queries.c pkg_str.c sqlite_callbacks.c selection.c \
 		pkg_check.c pkg_infos.c preferred.c
 # included from libinstall
-SRCS+=		automatic.c decompress.c dewey.c fexec.c global.c \
-		opattern.c pkgdb.c var.c
+SRCS+=		automatic.c dewey.c fexec.c global.c opattern.c pkgdb.c var.c
 # included from openssh
 SRCS+=		progressmeter.c
 

external/lib.h

@@ -441,9 +441,6 @@ lpkg_t *alloc_lpkg(const char *);
 lpkg_t *find_on_queue(lpkg_head_t *, const char *);
 void    free_lpkg(lpkg_t *);
 
-/* Extract input if compressed to NUL terminated buffer (not counted) */
-int decompress_buffer(const char *, size_t, char **, size_t *);
-
 /* Parse NUL terminated inputed, argument is strlen of the input */
 struct pkg_vulnerabilities *parse_pkg_vulnerabilities(const char *, size_t, int);
 /* Read pkg_vulnerabilities from file */