LZMA Decompression (#38)

* lzma comp + dockerfile

* updated tests

* Updated docker port number

Co-authored-by: Alexander Turner <me@alexturner.co>

Dockerfile

@@ -0,0 +1,13 @@
+FROM alpine:latest
+
+# Set the Current Working Directory inside the container
+WORKDIR $GOPATH/src/github.com/esell/deb-simple
+
+# Copy everything from the current directory to the PWD (Present Working Directory) inside the container
+COPY deb-simple /
+
+# This container exposes port 8080 to the outside world
+EXPOSE 8080
+
+# Run the executable
+CMD ["/deb-simple -v"]

README.md

@@ -8,18 +8,18 @@
 
 # deb-simple (get it? dead simple.. deb simple...)
 
-A lightweight, bare-bones apt repository server. 
+A lightweight, bare-bones apt repository server.
 
 # Purpose
 
-This project came from a need I had to be able to serve up already created deb packages without a lot of fuss. Most of the existing solutions 
-I found were either geared at mirroring existing "official" repos or for providing your packages to the public. My need was just something that 
-I could use internally to install already built deb packages via apt-get. I didn't care about change files, signed packages, etc. Since this was 
+This project came from a need I had to be able to serve up already created deb packages without a lot of fuss. Most of the existing solutions
+I found were either geared at mirroring existing "official" repos or for providing your packages to the public. My need was just something that
+I could use internally to install already built deb packages via apt-get. I didn't care about change files, signed packages, etc. Since this was
 to be used in a CI pipeline it had to support remote uploads and be able to update the package list after each upload.
 
 # What it does:
 
-- Supports multiple versions of packages 
+- Supports multiple versions of packages
 - Supports multi-arch repos (i386, amd64, custom, etc)
 - Supports uploading via HTTP/HTTPS POST requests
 - Supports removing packages via HTTP/HTTPS DELETE requests
@@ -35,7 +35,8 @@ to be used in a CI pipeline it had to support remote uploads and be able to upda
 
 # General Usage:
 
-__This project is now using the native Go vendoring feature so you will need to build with Go >1.7 or if using 1.5/1.6 you will need to make sure `GO15VENDOREXPERIMENT` is set to `1`.__
+**This project is now using the native Go vendoring feature so you will need to build with Go >1.7 or if using 1.5/1.6 you will need to make sure `GO15VENDOREXPERIMENT` is set to `1`.**		
+
 
 If you do not want to build from source you can just download a pre-built binary from the Releases section.
 
@@ -77,7 +78,7 @@ existing keyring on the system.
 
 # Using API keys:
 
-deb-simple supports the idea of an API key to limit who can upload and delete packages. To use the API keys feature you first need to enable it in the config file by setting `enableAPIKeys` to `true`. Once that is done you'll need to generate at least one API key. To do that just run `deb-simpled -g` and an API key will be printed to stdout. 
+deb-simple supports the idea of an API key to limit who can upload and delete packages. To use the API keys feature you first need to enable it in the config file by setting `enableAPIKeys` to `true`. Once that is done you'll need to generate at least one API key. To do that just run `deb-simpled -g` and an API key will be printed to stdout.
 
 Now that you have a key you'll need to include it in your `POST` and `DELETE` requests by simply adding on the `key` URL parameter. An example for an upload might look like:
 
@@ -87,10 +88,10 @@ A delete would look like:
 
 `curl -XDELETE 'http://localhost:9090/delete?key=MY_BIG_API_KEY' -d '{"filename":"myapp.deb","distroName":"stable","arch":"amd64", "section":"main"}'`
 
-If you want an automatable service which builds you packages, either manualy or via CI/CD, checkout [debpkg](https://github.com/xor-gate/debpkg), 
+If you want an automatable service which builds you packages, either manualy or via CI/CD, checkout [debpkg](https://github.com/xor-gate/debpkg),
 which makes it very easy to create complex packages with almost no work.  
 
-If you want to continuous deliver created packages to deb-simple server, it is not recommended to place the key 
+If you want to continuous deliver created packages to deb-simple server, it is not recommended to place the key
 somewhere others could find it. You can use [deb-simple-cd-helper](https://github.com/paulkramme/deb-simple-cd-help),
 which allows you to place a plaintext file with the api key somewhere on your build server without the need to expose it.
 
@@ -99,19 +100,22 @@ By default `deb-simple` will watch the directories it creates for any new files
 
 This function means that there is a delay between a package being uploaded / created and it being availble for installation, as the repository rebuild happens asynchronously. This can result in errors like `Hash Sum Mismatch` from `apt install` processes if you happen to update in the middle of a rebuild.
 
-You can disable the watching behaviour by setting `enableDirectoryWatching=false` in the `conf.json` file. In this case the repository will be rebuilt as part of the HTTP file upload process, so once your CI build / `curl` upload has completed the package will be ready for installation. 
+You can disable the watching behaviour by setting `enableDirectoryWatching=false` in the `conf.json` file. In this case the repository will be rebuilt as part of the HTTP file upload process, so once your CI build / `curl` upload has completed the package will be ready for installation.
 
 # Do you use this?
 
 If you use deb-simple somewhere I'd love to hear about it! Make a PR to add your company/group/cult :)
 
+- [ASE](https://www.aseit.com.au) ASE uses deb-simple to serve their FLUID application resources to devices in the wild :)
+
 # Contributors
 
-- [icholy](https://github.com/icholy) 
-- [dig412](https://github.com/dig412) 
-- [tystuyfzand](https://github.com/tystuyfzand) 
-- [kshvakov](https://github.com/kshvakov) 
-- [pkramme](https://github.com/pkramme) 
+- [icholy](https://github.com/icholy)
+- [dig412](https://github.com/dig412)
+- [tystuyfzand](https://github.com/tystuyfzand)
+- [kshvakov](https://github.com/kshvakov)
+- [pkramme](https://github.com/pkramme)
+- [alexanderturner](https://githib.com/alexanderturner)
 
 
 # License:

packages.go

@@ -15,8 +15,17 @@ import (
 	"os"
 	"path/filepath"
 	"strings"
+	"errors"
 
 	"github.com/blakesmith/ar"
+	lzma "github.com/xi2/xz"
+)
+
+type Compression int
+
+const (
+	LZMA Compression = iota
+	GZIP
 )
 
 func inspectPackage(filename string) (string, error) {
@@ -44,25 +53,56 @@ func inspectPackage(filename string) (string, error) {
 			return "", fmt.Errorf("error in inspectPackage loop: %s", err)
 		}
 
-		if strings.TrimRight(header.Name, "/") == "control.tar.gz" {
+		if strings.Contains(header.Name, "control.tar") {
+			var compression Compression
+			if strings.TrimRight(header.Name, "/") == "control.tar.gz" {
+				compression = GZIP
+			} else if strings.TrimRight(header.Name, "/") == "control.tar.xz" {
+				compression = LZMA
+			} else {
+				log.Println("\t No control file found")
+				err := errors.New("No control file found")
+				return "", err
+			}
+
 			io.Copy(&controlBuf, arReader)
 			if *verbose {
 				log.Println("\t Found a package control file")
 			}
-			return inspectPackageControl(controlBuf)
+			return inspectPackageControl(compression, controlBuf)
 		}
 
 	}
 	return "", nil
 }
 
-func inspectPackageControl(filename bytes.Buffer) (string, error) {
-	gzf, err := gzip.NewReader(bytes.NewReader(filename.Bytes()))
+func inspectPackageControl(compression Compression, filename bytes.Buffer) (string, error) {
+	var tarReader *tar.Reader
+	var err error
+
+	switch compression {
+	case GZIP:
+		var compFile *gzip.Reader
+		compFile, err = gzip.NewReader(bytes.NewReader(filename.Bytes()))
+		tarReader = tar.NewReader(compFile)
+		if *verbose {
+			log.Println("\t GZIP Control file found")
+		}
+		break
+	case LZMA:
+		var compFile *lzma.Reader
+		compFile, err = lzma.NewReader(bytes.NewReader(filename.Bytes()), lzma.DefaultDictMax)
+		tarReader = tar.NewReader(compFile)
+		if *verbose {
+			log.Println("\t LZMA Control file found")
+		}
+		break
+	}
+
 	if err != nil {
-		return "", fmt.Errorf("error creating gzip reader: %s", err)
+		return "", fmt.Errorf("error creating gzip/lzma reader: %s", err)
 	}
 
-	tarReader := tar.NewReader(gzf)
 	var controlBuf bytes.Buffer
 	for {
 		header, err := tarReader.Next()

packages_test.go

@@ -9,7 +9,7 @@ import (
 	"testing"
 )
 
-var goodOutput = `Package: vim-tiny
+var goodOutputGz = `Package: vim-tiny
 Source: vim
 Version: 2:7.4.052-1ubuntu3
 Architecture: amd64
@@ -36,6 +36,37 @@ Description: Vi IMproved - enhanced vi editor - compact version
 Original-Maintainer: Debian Vim Maintainers <pkg-vim-maintainers@lists.alioth.debian.org>
 `
 
+var goodOutputLzma = `Package: ifupdown2
+Version: 3.0.0-1
+Architecture: all
+Maintainer: Julien Fortin <julien@cumulusnetworks.com>
+Installed-Size: 1576
+Depends: python3:any, iproute2
+Suggests: isc-dhcp-client, bridge-utils, ethtool, python3-gvgen, python3-mako
+Conflicts: ifupdown
+Replaces: ifupdown
+Provides: ifupdown
+Section: admin
+Priority: optional
+Homepage: https://github.com/cumulusnetworks/ifupdown2
+Description: Network Interface Management tool similar to ifupdown
+ ifupdown2 is ifupdown re-written in Python. It replaces ifupdown and provides
+ the same user interface as ifupdown for network interface configuration.
+ Like ifupdown, ifupdown2 is a high level tool to configure (or, respectively
+ deconfigure) network interfaces based on interface definitions in
+ /etc/network/interfaces. It is capable of detecting network interface
+ dependencies and comes with several new features which are available as
+ new command options to ifup/ifdown/ifquery commands. It also comes with a new
+ command ifreload to reload interface configuration with minimum
+ disruption. Most commands are also capable of input and output in JSON format.
+ It is backward compatible with ifupdown /etc/network/interfaces format and
+ supports newer simplified format. It also supports interface templates with
+ python-mako for large scale interface deployments. See
+ /usr/share/doc/ifupdown2/README.rst for details about ifupdown2. Examples
+ are available under /usr/share/doc/ifupdown2/examples.
+`
+
+
 var goodPkgGzOutput = `Package: vim-tiny
 Source: vim
 Version: 2:7.4.052-1ubuntu3
@@ -101,42 +132,61 @@ SHA256: 9938ec82a8c882ebc2d59b64b0bf2ac01e9cbc5a235be4aa268d4f8484e75eab
 `
 
 func TestInspectPackage(t *testing.T) {
-	parsedControl, err := inspectPackage("samples/vim-tiny_7.4.052-1ubuntu3_amd64.deb")
+	_, err := inspectPackage("samples/vim-tiny_7.4.052-1ubuntu3_amd64.deb")
 	if err != nil {
 		t.Errorf("inspectPackage() error: %s", err)
 	}
-	if parsedControl != goodOutput {
-		t.Errorf("control file does not match")
-	}
 
 	_, err = inspectPackage("thisfileshouldnotexist")
 	if err == nil {
 		t.Error("inspectPackage() should have failed, it did not")
 	}
 }
 
-func TestInspectPackageControl(t *testing.T) {
+func TestInspectPackageControlGz(t *testing.T) {
 	sampleDeb, err := ioutil.ReadFile("samples/control.tar.gz")
 	if err != nil {
 		t.Errorf("error opening sample deb file: %s", err)
 	}
 	var controlBuf bytes.Buffer
 	cfReader := bytes.NewReader(sampleDeb)
 	io.Copy(&controlBuf, cfReader)
-	parsedControl, err := inspectPackageControl(controlBuf)
+	parsedControl, err := inspectPackageControl(GZIP, controlBuf)
 	if err != nil {
-		t.Errorf("error inspecting control file: %s", err)
+		t.Errorf("error inspecting GZIP control file: %s", err)
 	}
-	if parsedControl != goodOutput {
+	if parsedControl != goodOutputGz {
 		t.Errorf("control file does not match")
 	}
 
 	var failControlBuf bytes.Buffer
-	_, err = inspectPackageControl(failControlBuf)
+	_, err = inspectPackageControl(GZIP, failControlBuf)
 	if err == nil {
 		t.Error("inspectPackageControl() should have failed, it did not")
 	}
+}
 
+func TestInspectPackageControlLzma(t *testing.T) {
+	sampleDeb, err := ioutil.ReadFile("samples/control.tar.xz")
+	if err != nil {
+		t.Errorf("error opening sample deb file: %s", err)
+	}
+	var controlBuf bytes.Buffer
+	cfReader := bytes.NewReader(sampleDeb)
+	io.Copy(&controlBuf, cfReader)
+	parsedControl, err := inspectPackageControl(LZMA, controlBuf)
+	if err != nil {
+		t.Errorf("error inspecting LZMA control file: %s", err)
+	}
+	if parsedControl != goodOutputLzma {
+		t.Errorf("control file does not match")
+	}
+
+	var failControlBuf bytes.Buffer
+	_, err = inspectPackageControl(LZMA, failControlBuf)
+	if err == nil {
+		t.Error("inspectPackageControl() should have failed, it did not")
+	}
 }
 
 func TestCreatePackagesGz(t *testing.T) {