-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
5 changed files
with
269 additions
and
3 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
# Getting started | ||
|
||
This package allows to create `tar` files and compress any file via `gzip` all in memory. This means that you don't have to have the source files written to disk to compress them. | ||
|
||
This package also work lazily, meaning that you never have to load a whole file in memory allowing you to work with files that may not fit in memory. | ||
|
||
Combining this package with the rest of the [Innmind ecosystem](https://github.com/innmind/) unlocks opportunities that weren't possible previously (or at least very hard to achieve). Here are some use cases: | ||
|
||
- [Creating a backup from different sources](use_cases/backup.md) | ||
- [Storing compressed files](use_cases/compressed_at_rest.md) | ||
- [Sending compressed files through HTTP](use_cases/http.md) | ||
|
||
> **Note** | ||
> All use cases use the [`innmind/operating-system`](https://packagist.org/packages/innmind/operating-system) package. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,76 @@ | ||
# Creating a backup from different sources | ||
|
||
```php | ||
use Innmind\OperatingSystem\Factory; | ||
use Formal\AccessLayer\Query\SQL; | ||
use Innmind\Filesystem\{ | ||
Name, | ||
File\, | ||
File\Content\Line, | ||
Directory, | ||
}; | ||
use Innmind\MediaType\MediaType; | ||
use Innmind\Url\{ | ||
Url, | ||
Path, | ||
}; | ||
use Innmind\Immutable\Str; | ||
use Innmind\Encoding\{ | ||
Gzip, | ||
Tar, | ||
}; | ||
|
||
$os = Factory::build(); | ||
|
||
$data = $os | ||
->filesystem() | ||
->mount(Path::of('path/to/stored/data')) | ||
->root() | ||
->rename(Name::of('data')); | ||
$sql = $os->remote()->sql(Url::of('mysql://{user}:{password}@localhost:3306/{database}')); | ||
$users = $sql(SQL::onDemand('SELECT * FROM users')) | ||
->map(static fn($row) => \implode( | ||
"\t", | ||
\array_values($row->toArray()), | ||
)) | ||
->map(Str::of(...)) | ||
->map(Line::of(...)); | ||
|
||
$archive = Gzip::compress()( | ||
Tar::encode($os->clock())( | ||
Directory::named('archive') | ||
->add($data) | ||
->add(File::named( | ||
'users.tsv', | ||
$users, | ||
MediaType::of('text/tab-separated-values'), | ||
)), | ||
), | ||
); | ||
``` | ||
|
||
Up to this point `$archive` represents a file named `archive.tar.gz` but no real operation has been done. For the real compression to happen you need to _unwrap_ the file either by persisting to the filesystem, sending it through HTTP/AMQP or returning it as an HTTP response. Here's an example of a simple file responding to an HTTP request: | ||
|
||
```php | ||
use Innmind\Http\{ | ||
ResponseSender, | ||
Response, | ||
Response\StatusCode, | ||
ProtocolVersion, | ||
Headers, | ||
Header\ContentType, | ||
}; | ||
|
||
$archive = /* see above */; | ||
|
||
(new ResponseSender($os->clock))(Response::of( | ||
StatusCode::ok, | ||
ProtocolVersion::v11, | ||
Headers::of( | ||
ContentType::of('application', 'octet-stream') | ||
), | ||
$archive->content(), | ||
)); | ||
``` | ||
|
||
And that's it ! `ResponseSender` will stream the archive chunk by chunk. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,117 @@ | ||
# Storing compressed files | ||
|
||
In many applications we allow our users to upload files. Storing them can take up a lot of space. | ||
|
||
Instead of storing these files as is, we could store them gzipped to save space and return them gzipped or uncompressed depending of the supported features. | ||
|
||
Here's an example: | ||
|
||
```php | ||
use Innmind\OperatingSystem\Factory; | ||
use Innmind\Http\Factory\ServerRequest\ServerRequestFactory; | ||
use Innmind\Filesystem\Name; | ||
use Innmind\Encoding\Gzip; | ||
use Innmind\Url\Path; | ||
use Innmind\Http\{ | ||
Response, | ||
Response\StatusCode, | ||
ResponseSender, | ||
}; | ||
|
||
$os = Factory::build(); | ||
$serverRequest = ServerRequestFactory::default($os->clock())(); | ||
|
||
$response = $serverRequest | ||
->files() | ||
->under('tsv') | ||
->get('users') | ||
->map(static fn($file) => $file->rename(Name::of('users.tsv')) | ||
->map(Gzip::compress()) | ||
->map( | ||
static fn($file) => $os | ||
->filesystem() | ||
->mount(Path::of('path/to/stored/data/')) | ||
->add($file)), | ||
) | ||
->match( | ||
static fn() => Response::of( | ||
StatusCode::created, | ||
$serverRequest->protocolVersion(), | ||
), | ||
static fn() => Response::of( | ||
StatusCode::badRequest, | ||
$serverRequest->protocolVersion(), | ||
), | ||
); | ||
|
||
(new ResponseSender($os->clock()))($response); | ||
``` | ||
|
||
This code will take any file uploaded in the key `tsv[users]`, gzip it and write it in the `path/to/stored/data/` directory under the name `users.tsv.gz` (`Gzip::compress()` automatically add the suffix `.gz`) and return a `201` HTTP response. If the upload failed it will return a `400` response. | ||
|
||
And for the code streaming this file: | ||
|
||
```php | ||
use Innmind\OperatingSystem\Factory; | ||
use Innmind\Http\Factory\ServerRequest\ServerRequestFactory; | ||
use Innmind\Filesystem\Name; | ||
use Innmind\Encoding\Gzip; | ||
use Innmind\Url\Path; | ||
use Innmind\Http\{ | ||
Response, | ||
Response\StatusCode, | ||
ResponseSender, | ||
Headers, | ||
Header\ContentType, | ||
Header\ContentEncoding, | ||
}; | ||
|
||
$os = Factory::build(); | ||
$serverRequest = ServerRequestFactory::default($os->clock())(); | ||
|
||
$acceptGzip = $serverRequest | ||
->headers() | ||
->get('accept-encoding') | ||
->map(static fn($header): bool => $header->values()->any( | ||
static fn($value) => $value->toString() === 'gzip', | ||
)) | ||
->match( | ||
static fn(bool $accept): bool => $accept, | ||
static fn() => false, | ||
); | ||
|
||
$response = $os | ||
->filesystem() | ||
->mount(Path::of('path/to/stored/data/')) | ||
->get(Name::of('users.tsv.gz')) | ||
->map(static fn($file) => match ($acceptGzip) { | ||
true => Response::of( | ||
StatusCode::ok, | ||
$serverRequest->protocolVersion(), | ||
Headers::of( | ||
ContentEncoding::of('gzip'), | ||
ContentType::of('text', 'tab-separated-values'), | ||
), | ||
$file->content(), | ||
), | ||
false => Response::of( | ||
StatusCode::ok, | ||
$serverRequest->protocolVersion(), | ||
Headers::of( | ||
ContentType::of('text', 'tab-separated-values'), | ||
), | ||
Gzip::decompress()($file->content()), | ||
), | ||
}) | ||
->match( | ||
static fn($response) => $response, | ||
static fn() => Response::of( | ||
StatusCode::notFound, | ||
$serverRequest->protocolVersion(), | ||
), | ||
); | ||
|
||
(new ResponseSender($os->clock()))($response); | ||
``` | ||
|
||
Here we try to load the `users.tsv.gz` file, we check if the caller accepts a gzipped content, if so we return the file as is via a `200` HTTP response and if not we decompress the file and return it. And if the file doesn't exist we return a `400` response. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,40 @@ | ||
# Sending compressed files through HTTP | ||
|
||
```php | ||
use Innmind\OperatingSystem\Factory; | ||
use Innmind\Filesystem\Name; | ||
use Innmind\Encoding\Gzip; | ||
use Innmind\Http\{ | ||
Request, | ||
Method, | ||
ProtocolVersion, | ||
Headers, | ||
Header\ContentEncoding, | ||
}; | ||
use Innmind\Url\{ | ||
Url, | ||
Path, | ||
}; | ||
|
||
$os = Factory::build(); | ||
$http = $os->remote()->http(); | ||
$os | ||
->filesystem() | ||
->mount(Path::of('path/to/stored/data/')) | ||
->get(Name::of('somefile.txt')) | ||
->map(Gzip::compress()) | ||
->match( | ||
static fn($file) => $http(Request::of( | ||
Url::of('https://some-app.tld/upload'), | ||
Method::post, | ||
ProtocolVersion::v11, | ||
Headers::of( | ||
ContentEncoding::of('gzip'), | ||
), | ||
$file->content(), | ||
)), | ||
static fn() => null, | ||
); | ||
``` | ||
|
||
If `somefile.txt` exists then it is gzipped and then sent to `https://some-app.tld/upload`. If the file doesn't exist then nothing is done. |