Phar Building

[defuse]

Supersedes #258.

WARNING: I force pushed

[defuse]

@paragonie-scott: Does this change look good to you?

[defuse]

The only thing I'm unsure of is the change to using classmap in composer.json. Do we even need to keep autoload.php anymore? If we're keeping autoload.php so that people can require it from a git clone of the repository, then I want composer to use that just so we'll be notified earlier in case it breaks.

[defuse]

Yeah, I'm going to revert this one change.

[defuse]

Oh wait, we want to get rid of random_compat, so I'll delete that and autoload.php and keep this change.

[glensc]

yes, it was all needed, the cleanups were listed as todo items in #258, but somewhy it was closed pointing here, but no back reference to the previous task. i was waiting for base grounds to be settled before actual removing outdated artifacts.

in short: autoload.php file is no longer needed, but change from "files" to "classmap" is needed.

[defuse]

Need to add @paragonie-scott to the authors list in composer.json.

[defuse]

@paragonie-scott: I added you, please check if I got the right URL and email address :)

[paragonie-scott]

Yep, that one works. :)

[glensc]

seems you broke support for box.phar and box executables (same for other tools). most people have .phar extension when they download stuff, but i also have distro package for composer.

[defuse]

Yeah, I added corresponding instructions to the developer docs that they have to be called composer and box in your path. I prefer that over adding complexity to the Makefile, since I'm the only one who should be running it anyway.

[glensc]

btw, does the WARNING: I force pushed note mean you removed those big images from v2.0 branch history?

[defuse]

@glensc: No, it just means I did a rebase of this branch and did a force push (I won't be removing the big images).

[defuse]

This can be removed now.

[glensc]

also from #258 tasklist, remained unresolved: compression to use for phar (currently GZ)

i've heard complaints of having compressed phar being slow
eventum/eventum@3ef291e

you should maybe benchmark this, inspect does the slowness comes due php engine extracting contents to temp dir or just cpu cycles wasted from gzuncompress. also i personally would like to know how .phar and opcache engines (xcache, zendoptimizer, apc) work with phar vs plain filesystem objects.

[paragonie-scott]

you should maybe benchmark this, inspect does the slowness comes due php engine extracting contents to temp dir or just cpu cycles wasted from gzuncompress. also i personally would like to know how .phar and opcache engines (xcache, zendoptimizer, apc) work with phar vs plain filesystem objects.

Dissenting opinion: That's a lot of work digging into PHP's internals, which might block the release of this library, which could be easily solved by "turn compression off; the deliverable is under 1 MB anyway".

[defuse]

Yeah, let's just turn compression off.

[defuse]

Also TODO: Get travis-CI to build the .phar

[defuse]

Everything's done, just need to understand what those last lines in box.json mean :)

[glensc]

Compactors strip the sourcecode before inserting to .phar. the php stripper works in a way that removes comments but preserves line numbers (so the errors would still be having meaningful line numbers).

extract the phar and see:

$ mkdir tmp
$ cd tmp/
$ phar extract -f ../defuse-crypto.phar 

[defuse]

Thanks, okay this LGTM!

[glensc]

also you can remove "exclude": "other" for "vendor/composer", there's no "other" subdir in vendor/composer like there is in vendor/paragonie/random_compat

[defuse]

(I tried removing the compactors since it's possible it not being implemented correctly could add errors to the code, but it doesn't seem to work, the result was the same whether those lines were there or they weren't).

[glensc]

the compactors come from composer source i'm pretty sure of that

[defuse]

Ok, after these tests pass I'll merge it. Thanks!

[glensc]

compactor is token based, that's the right way to do it
https://github.com/box-project/box2-lib/blob/master/src/lib/Herrera/Box/Compactor/Php.php

[paragonie-scott]

The compactors won't affect reproducibility verification with Pharaoh, right?

[glensc]

compactor just strips the comments. if you want to diff comments, then it's yeah affected. otherwise executable code stays the same. also, composer.phar has always been stripped like this.

[defuse]

It looks like it messes with the whitespace too (which is where I'm afraid subtle bugs in its compacting algorithm could break things).

[defuse]

But yeah hopefully it's deterministic and shouldn't affect Pharaoh.

[glensc]

of course it's deterministic, there's no side effects of tokenizing static files.

[paragonie-scott]

Since it's not compressed, I can probably re-audit the .phar (e.g. with a hex editor) if we have any concerns about backdoored Phar builders.

[glensc]

even if it was compressed, you can decompress it