Skip to content

Commit

Permalink
add documentation
Browse files Browse the repository at this point in the history
  • Loading branch information
Baptouuuu committed Oct 29, 2023
1 parent 556b6b6 commit bd8bab0
Show file tree
Hide file tree
Showing 5 changed files with 269 additions and 3 deletions.
25 changes: 22 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,13 @@
# encoding
# Encoding

[![Build Status](https://github.com/innmind/encoding/workflows/CI/badge.svg?branch=master)](https://github.com/innmind/encoding/actions?query=workflow%3ACI)
[![codecov](https://codecov.io/gh/innmind/encoding/branch/develop/graph/badge.svg)](https://codecov.io/gh/innmind/encoding)
[![Type Coverage](https://shepherd.dev/github/innmind/encoding/coverage.svg)](https://shepherd.dev/github/innmind/encoding)

Description
This packages allows to encode and compress files and directories without the need for them to be written to the filesystem and never loaded entirely in memory.

> **Note**
> Each file contained in a `tar` file can't exceed an 8Go size.
## Installation

Expand All @@ -14,4 +17,20 @@ composer require innmind/encoding

## Usage

Todo
Take a look at the [documentation](documentation/README.md) for a more in-depth understanding of the possibilities.

### Creating an archive of a directory

```php
$adapter = Adapter::mount(Path::of('some/directory/'));
$tar = $adapter
->get(Name::of('data'))
->map(Tar::encode(new Earth\Clock))
->map(Gzip::compress())
->match(
static fn($file) => $file,
static fn() => null,
);
```

Here `$tar` represents a `.tar.gz` file containing all the files and directories from `sime/directory/data/`, unless the `data` doesn't exist then it is `null`.
14 changes: 14 additions & 0 deletions documentation/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Getting started

This package allows to create `tar` files and compress any file via `gzip` all in memory. This means that you don't have to have the source files written to disk to compress them.

This package also work lazily, meaning that you never have to load a whole file in memory allowing you to work with files that may not fit in memory.

Combining this package with the rest of the [Innmind ecosystem](https://github.com/innmind/) unlocks opportunities that weren't possible previously (or at least very hard to achieve). Here are some use cases:

- [Creating a backup from different sources](use_cases/backup.md)
- [Storing compressed files](use_cases/compressed_at_rest.md)
- [Sending compressed files through HTTP](use_cases/http.md)

> **Note**
> All use cases use the [`innmind/operating-system`](https://packagist.org/packages/innmind/operating-system) package.
76 changes: 76 additions & 0 deletions documentation/use_cases/backup.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
# Creating a backup from different sources

```php
use Innmind\OperatingSystem\Factory;
use Formal\AccessLayer\Query\SQL;
use Innmind\Filesystem\{
Name,
File\,
File\Content\Line,
Directory,
};
use Innmind\MediaType\MediaType;
use Innmind\Url\{
Url,
Path,
};
use Innmind\Immutable\Str;
use Innmind\Encoding\{
Gzip,
Tar,
};

$os = Factory::build();

$data = $os
->filesystem()
->mount(Path::of('path/to/stored/data'))
->root()
->rename(Name::of('data'));
$sql = $os->remote()->sql(Url::of('mysql://{user}:{password}@localhost:3306/{database}'));
$users = $sql(SQL::onDemand('SELECT * FROM users'))
->map(static fn($row) => \implode(
"\t",
\array_values($row->toArray()),
))
->map(Str::of(...))
->map(Line::of(...));

$archive = Gzip::compress()(
Tar::encode($os->clock())(
Directory::named('archive')
->add($data)
->add(File::named(
'users.tsv',
$users,
MediaType::of('text/tab-separated-values'),
)),
),
);
```

Up to this point `$archive` represents a file named `archive.tar.gz` but no real operation has been done. For the real compression to happen you need to _unwrap_ the file either by persisting to the filesystem, sending it through HTTP/AMQP or returning it as an HTTP response. Here's an example of a simple file responding to an HTTP request:

```php
use Innmind\Http\{
ResponseSender,
Response,
Response\StatusCode,
ProtocolVersion,
Headers,
Header\ContentType,
};

$archive = /* see above */;

(new ResponseSender($os->clock))(Response::of(
StatusCode::ok,
ProtocolVersion::v11,
Headers::of(
ContentType::of('application', 'octet-stream')
),
$archive->content(),
));
```

And that's it ! `ResponseSender` will stream the archive chunk by chunk.
117 changes: 117 additions & 0 deletions documentation/use_cases/compressed_at_rest.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
# Storing compressed files

In many applications we allow our users to upload files. Storing them can take up a lot of space.

Instead of storing these files as is, we could store them gzipped to save space and return them gzipped or uncompressed depending of the supported features.

Here's an example:

```php
use Innmind\OperatingSystem\Factory;
use Innmind\Http\Factory\ServerRequest\ServerRequestFactory;
use Innmind\Filesystem\Name;
use Innmind\Encoding\Gzip;
use Innmind\Url\Path;
use Innmind\Http\{
Response,
Response\StatusCode,
ResponseSender,
};

$os = Factory::build();
$serverRequest = ServerRequestFactory::default($os->clock())();

$response = $serverRequest
->files()
->under('tsv')
->get('users')
->map(static fn($file) => $file->rename(Name::of('users.tsv'))
->map(Gzip::compress())
->map(
static fn($file) => $os
->filesystem()
->mount(Path::of('path/to/stored/data/'))
->add($file)),
)
->match(
static fn() => Response::of(
StatusCode::created,
$serverRequest->protocolVersion(),
),
static fn() => Response::of(
StatusCode::badRequest,
$serverRequest->protocolVersion(),
),
);

(new ResponseSender($os->clock()))($response);
```

This code will take any file uploaded in the key `tsv[users]`, gzip it and write it in the `path/to/stored/data/` directory under the name `users.tsv.gz` (`Gzip::compress()` automatically add the suffix `.gz`) and return a `201` HTTP response. If the upload failed it will return a `400` response.

And for the code streaming this file:

```php
use Innmind\OperatingSystem\Factory;
use Innmind\Http\Factory\ServerRequest\ServerRequestFactory;
use Innmind\Filesystem\Name;
use Innmind\Encoding\Gzip;
use Innmind\Url\Path;
use Innmind\Http\{
Response,
Response\StatusCode,
ResponseSender,
Headers,
Header\ContentType,
Header\ContentEncoding,
};

$os = Factory::build();
$serverRequest = ServerRequestFactory::default($os->clock())();

$acceptGzip = $serverRequest
->headers()
->get('accept-encoding')
->map(static fn($header): bool => $header->values()->any(
static fn($value) => $value->toString() === 'gzip',
))
->match(
static fn(bool $accept): bool => $accept,
static fn() => false,
);

$response = $os
->filesystem()
->mount(Path::of('path/to/stored/data/'))
->get(Name::of('users.tsv.gz'))
->map(static fn($file) => match ($acceptGzip) {
true => Response::of(
StatusCode::ok,
$serverRequest->protocolVersion(),
Headers::of(
ContentEncoding::of('gzip'),
ContentType::of('text', 'tab-separated-values'),
),
$file->content(),
),
false => Response::of(
StatusCode::ok,
$serverRequest->protocolVersion(),
Headers::of(
ContentType::of('text', 'tab-separated-values'),
),
Gzip::decompress()($file->content()),
),
})
->match(
static fn($response) => $response,
static fn() => Response::of(
StatusCode::notFound,
$serverRequest->protocolVersion(),
),
);

(new ResponseSender($os->clock()))($response);
```

Here we try to load the `users.tsv.gz` file, we check if the caller accepts a gzipped content, if so we return the file as is via a `200` HTTP response and if not we decompress the file and return it. And if the file doesn't exist we return a `400` response.
40 changes: 40 additions & 0 deletions documentation/use_cases/http.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# Sending compressed files through HTTP

```php
use Innmind\OperatingSystem\Factory;
use Innmind\Filesystem\Name;
use Innmind\Encoding\Gzip;
use Innmind\Http\{
Request,
Method,
ProtocolVersion,
Headers,
Header\ContentEncoding,
};
use Innmind\Url\{
Url,
Path,
};

$os = Factory::build();
$http = $os->remote()->http();
$os
->filesystem()
->mount(Path::of('path/to/stored/data/'))
->get(Name::of('somefile.txt'))
->map(Gzip::compress())
->match(
static fn($file) => $http(Request::of(
Url::of('https://some-app.tld/upload'),
Method::post,
ProtocolVersion::v11,
Headers::of(
ContentEncoding::of('gzip'),
),
$file->content(),
)),
static fn() => null,
);
```

If `somefile.txt` exists then it is gzipped and then sent to `https://some-app.tld/upload`. If the file doesn't exist then nothing is done.

0 comments on commit bd8bab0

Please sign in to comment.