[WIP] Easy User-land CSPRNG #1119

SammyK · 2015-02-24T22:48:22Z

Submitting as a PR per @nikic's recommendation.

RFC can be found here.

lt · 2015-02-14T12:47:09Z

I wonder about the usefulness of this function. This can be easily achieved with random_bytes and bin2hex, so I feel like this is unnecessary duplication of functionality for the sake of one function call.

SammyK · 2015-02-16T21:20:15Z

Yeah, I'm wouldn't put up too much of a fight on this one, but 100% of the times I've ever needed random bytes it has always been in the form of hex or int. I'm thinking most user-land kids are in the same boat. :)

I noticed that the hex table for bin2hex() is just a char of 16 as opposed to the larger base64_encode() or session hex tables. Would it be considered more random with a bigger hex table?

SammyK · 2015-02-17T06:56:12Z

So I've been doing a bunch of research on binary to hex conversions and learned a lot. Since hex (base 16) is just a different representation of the same data in binary (base 2), it wouldn't necessarily add more randomness to convert the data to Base32 or Base64. But at the end of the day, the common user-land PHP developer won't know about these things in too much detail (as I didn't before tonight!) But what the user-land dev is thinking, "I need a random alpha-numeric string", so hex should suffice.

I've also be researching how other languages do random like Ruby, Go and Python. They all look for /dev/urandom or the CryptGenRandom on Windows.

BTW: I've seen most implementations of a unique ID in other languages make use of the CSPRNG like in Python. I think we should do the same with uniqid(). :)

lt · 2015-02-17T09:12:41Z

Good job doing the research (I feel almost proud! :)).

You're absolutely right, hex or base64 doesn't add anything. In fact it takes away entropy, and we're not talking about small amounts here.

So lets take hex for example. Conversion to hex literally takes an 8 bit representation of data, and turns it into a 4 bit representation of data (used as an index in a character table). So how much entropy do we really lose? Maths time (I'm thinking free-form here, and I'm not a mathematician, but hopefully this is along the right lines!)

The entropy in a base-encoded string compared to the raw binary is: 1 / (2^{(8 - bits_per_byte ) * nbytes}). Just to test, plugging in the full 8 bits per byte we get 1 / 2⁰ which is 1/1, full entropy.

So for example a 16 byte (128 bit) piece of keying material, keeping the input as hex rather than full binary, the key is 1 / 2^4*16 as strong. 1 / 2⁶⁴.

2⁶⁴ is a reasonably familiar number, being the number of combinations in a 64-bit unsigned integer. I can't remember it in full, but I can remember it's roughly 18.5 quintillion. A hex encoded string, for 16 bytes, is 18.5 quintillion times weaker than the full binary string. Incidentally, that is the same number of key combinations there are (2¹²⁸ / 2⁶⁴ == 2⁶⁴).

It's still going to take a long time to crack, but I have no idea how this affects attacks on ciphers that break X of so many rounds, if you can guarantee specific weaknesses in the key.

lt · 2015-02-17T09:13:27Z

I think migrating other parts of PHP to use a CSPRNG should be a separate PR.

uniqid() and session_create_id() are obvious candidates.

SammyK · 2015-02-17T18:34:40Z

Oh man! Math! I've got more research to do. :)

Yes, I agree with you on creating separate PR's for uniqid() and session_create_id(). Hopefully those will be easy to knock out after this one.

I should have a little bit more time on Thursday to convert some of this research into commits on this bad boy. :)

lt · 2015-02-14T12:48:31Z

Edit: - Hah, I see you already did this in your second commit :)

With your includes at the top:

#if PHP_WIN32
#include "win32/winutil.h"
#endif

Then use php_win32_get_random_bytes(unsigned char *buf, size_t len)
This already uses the CryptGenRandom windows API call, so we're all good here.

lt · 2015-02-14T12:55:34Z

This should go back to being zend_string and use bytes->val as your char *.
zend_string_alloc to create it and zend_string_release if you hit an error condition. Don't release and use RETURN_STR if you have success. This saves having to copy the string at the end of the function with RETVAL_STRINGL

lt · 2015-02-14T13:02:53Z

We need to be careful about uniformity in this function. If you use modulus to bring the number down to your upper bound and the divisor is not a power of 2, you can end up with some bias in the result which we obviously want to avoid with a CSPRNG.

I'll help with the implementation of fixing this.

SammyK · 2015-02-20T01:59:46Z

Should we remove the min option and just have max? None of the PRNG's in other languages I've reviewed have a min. Also, in Ruby the default functionality returns a float. Don't know if we want to explore that path as well.

lt · 2015-02-20T12:17:02Z

Remove min 👍
Return float 👎

SammyK · 2015-02-20T13:38:06Z

Sounds good!

lt · 2015-02-14T13:42:30Z

There's a few more options to investigate here. /dev/urandom is going to be "ok", but I also think it should be the last option. Where possible we should try and avoid opening file descriptors.

Off the top of my head, things to investigate (I can do all of this if you like):

What would be the security impact of having a user space arc4random implementation for Linux.
Where arc4random exists, and libc is shared amongst all processes, is the arc4random state shared or per process?
Attempt to use /dev/arandom if it exists and for some odd reason we don't have arc4random
Linux getrandom syscall (for kernel versions >= 3.17)

lt · 2015-02-14T13:49:43Z

To get a random number to work on do php_random_bytes(&number, sizeof(number));, so whatever source of random is picked as being the best for the platform is used.

First attempt at random_int(). Commented-out random_hex() until the other functions are rocking.

Changes from my fork of rand-bytes

lt · 2015-02-24T20:16:18Z

@Rican7 - Documentation issue in the RFC. The value in the implementation here is correct. I've let Sammy know.

SammyK · 2015-02-24T20:47:04Z

Fixed in the RFC.

Rican7 · 2015-02-24T22:18:13Z

👍

lt · 2015-02-24T20:20:36Z

@SammyK I derped in the error message, minimum definitely should not be greater than the maximum :)

lt · 2015-02-24T20:23:05Z

Perhaps we should make it so both min and max need to be specified to be valid? Doesn't really feel sensible to allow the min without a max. random_int(0) => 123456 // wat!

if (ZEND_NUM_ARGS() == 1) {
    php_error_docref(NULL, E_WARNING, "Error message of your choosing");
    RETURN_FALSE;
}

SammyK · 2015-02-24T20:47:52Z

Lol. Nice catch. :) I'll fix.

lt · 2015-02-24T20:20:52Z

This needs min putting back in.

nikic · 2015-02-24T23:12:37Z

ext/standard/random.c

+
+	if (php_random_bytes(bytes->val, size) == FAILURE) {
+		zend_string_release(bytes);
+		return;


Probably that should be RETURN_FALSE as well? Same in random_int.

Nice catch! I'll fix.

nikic · 2015-02-24T23:13:46Z

ext/standard/random.c

+
+static int php_random_bytes(void *bytes, size_t size)
+{
+	int n = 0;


Better would be ssize_t.

The declaration should also be moved into the while loop, otherwise this will likely generate a warning on BSD systems with arc4random.

laruence · 2015-02-25T09:03:18Z

ext/standard/random.c

+	zend_ulong result;
+
+	if (ZEND_NUM_ARGS() == 1) {
+		php_error_docref(NULL, E_WARNING, "A minimum and maximum value are expected, only minimum given");


since you already have LONG_MIN and LONG_MAX, so if only minimum value min is given. then simply assume it means min to LONG_MAX?

@laruence: I thought about the same thing but then I realize that reading:
random_int(10);
might be understood as 10 being the max.

hmm, that could be a doc issue.. but it's not a big deal :)

Oops - this is a dingleberry left over from when min and max were optional args. They are both required in the current spec that's being voted on. I'll remove this check. :)

laruence · 2015-02-25T09:04:24Z

ext/standard/random.c

+	}
+
+	if (min >= max) {
+		php_error_docref(NULL, E_WARNING, "Minimum value must be less than the maximum value");


maybe something like: 1st arg must be less than 2nd arg is more obvious for user?

lt · 2015-03-30T08:46:43Z

@sarciszewski This is something we have discussed, but decided to leave for future scope. getrandom (and getentropy on BSD) are typically used to seed RNGs, and are not designed for high throughput themselves.

If you compile against LibreSSL and have a Linux Kernel version > 3.17, you will be using getrandom today.

ircmaxell · 2015-04-09T16:55:52Z

The vote passed, but was never closed. This should be merged soon ;-)

nikic · 2015-04-09T16:59:09Z

ext/standard/random.c

+		RANDOM_G(fd) = fd;
+	}
+
+	ssize_t n = 0;


This needs to be moved into the while look (combined declaration and assignment) to conform with C89.

SammyK · 2015-04-09T18:33:12Z

Thanks for the push @ircmaxell! :) I'm coordinating with @lt to get this bad boy ready for merge. :)

lt · 2015-04-09T19:24:00Z

ext/standard/random.c

+Return an arbitrary pseudo-random integer */
+PHP_FUNCTION(random_int)
+{
+	zend_long min = ZEND_LONG_MIN;


@SammyK Since both parameters are required, we don't need these defaults any more.

SammyK · 2015-04-10T19:37:53Z

Hokay! @lt updated this PR based on feedback. I normalized the return values when errors happen & updated the tests & merged in the lest from master & fixed conflicts. Phew!

The RFC officially passed unanimously and this PR can be merged into master now. W00t! Not sure if @nikic is the one to do that or who we need to ping.

Thanks again @ircmaxell, @lt, @rdlowrey, and all the other kids who help me along on this one! 🐻 Let's do some more of these! :D

nikic · 2015-04-29T14:04:56Z

Zpp failures should always return null (what was implemented initially), unless there is some very strong reason to do otherwise. We usually only make exceptions to this for legacy functions.

nikic · 2015-05-09T20:40:26Z

Merged via #1268. Thanks everyone who worked on this!

SammyK · 2015-05-11T16:01:31Z

Yay! Thanks @nikic! :)

CodesInChaos · 2015-07-06T15:04:44Z

The documentation for random_int could use some improvements:
- Clarify the behaviour if min<=max is violated
- Specify if the bounds are inclusive or exclusive (looking at rand and mt_rand it's probably inclusive)
- It should mention that the distribution is uniform
Why does it return FALSE which can easily treated as 0 instead of something you can't ignore, like an exception when the RNG in unavailable?
What about a secure polyfill?

SammyK · 2015-07-06T17:54:30Z

Why does it return FALSE which can easily treated as 0 instead of something you can't ignore, like an exception when the RNG in unavailable?

Good point. I think a RuntimeException should be thrown when a proper source of random cannot be detected. That way when errors are turned off we won't have people using false/0 as their CSPRNG. :) Thoughts @lt?

SammyK · 2015-07-06T22:22:42Z

@lt: I just pushed up a branch with this change, can you take a look? :)

CodesInChaos · 2015-07-07T06:36:50Z

@SammyK What should happen if min > max? Also an exception?

lt · 2015-07-07T07:53:18Z

I think when we wrote this, exceptions in the engine hadn't yet passed.

I'd be happy with InvalidArgumentException for min>max and len<=0

CodesInChaos · 2015-07-07T10:43:07Z

Should len === 0 be considered invalid? I'd consider it valid and only throw an exception if len < 0.

tom-- · 2015-07-07T13:11:17Z

When someone tries to generate an empty string using the CSPRNG, I think it's likely a programming error (if it were my project, I'd define it as a programming error). So I think it is safer if PHP treats this case as invalid.

And, from the other point of view, I think PHP can reasonably reject bug reports that you can't generate empty strings with the CSPRNG, especially if the conditions for these exceptions are documented.

lt · 2015-07-07T13:40:16Z

If it's not a programming error, it's a stupid thing to do. Invalid all the way.

CodesInChaos · 2015-07-07T14:48:55Z

At least min == max on random_int shouldn't be an error, since that can be useful in practice. For example when picking the last card in a shuffling algorithm.

tom-- · 2015-07-07T14:54:52Z

@CodesInChaos I agree.

lt · 2015-07-07T16:16:56Z

I'll disagree with practically useful. But I'll agree that it should be allowed, in the "spirit of PHP"

If we're going to allow min == max it should be special cased, no point fetching from the rng for it.

    if (min >= max) {
        if (min == max) {
            RETURN_LONG(min);
        }

        zend_throw_exception(spl_ce_InvalidArgumentException, "Maximum value must not be less than the minimum value", 0);
        return;

    }

@SammyK perhaps do something like this for SPL exceptions:

php-src/ext/mysqli/mysqli.c

Line 606 in ed1b648

#ifdef HAVE_SPL

So it can fall back to a less specific exception if SPL is not included.

nikic · 2015-07-07T16:31:25Z

Please do the change to exceptions separately from the changing any error conditions.

SammyK · 2015-07-07T17:37:43Z

Sounds good. I'll submit 2 separate PR's - one for throwing general exceptions and one for new error conditions. Then I'll update the docs when they are merged and post here for another once-over to make sure the docs cover all the bases. :)

SammyK added 2 commits Feb 13, 2015

Got initial API added

f8d7aec

Got mostly working. I think Hex is broken. Need to add int.

26e4ed2

SammyK and others added 12 commits Feb 20, 2015

Updated to use zend_string but getting memory leaks. Doh!

3c5fcac

Got random_bytes() working again.

e96b077

First attempt at random_int(). Commented-out random_hex() until the other functions are rocking.

Fix random_int() checking size of wrong var.

aa0ca69

Got random_int() seemingly working thanks to @ircmaxell

b32e0d0

Make maximum argument to random_int() optional with default to INT_MAX.

2c659ed

Remove random_hex(). *sadface*

a1e6229

Detect presence of /dev/arandom

bbc9198

Tidy up `php_random_bytes` and add /dev/arandom

7a99db6

Ensure random_int() uses a uniform distribution

3d413ad

Allow full integer range from random_int()

513d5c9

Use arc4random where present

77f99cc

Merge pull request #1 from lt/rand-bytes

7ef5754

Changes from my fork of rand-bytes

SammyK added 2 commits Feb 24, 2015

Add tests

766ce0c

Fix wording in error message. Add check for max value.

Loading status checks…

99e36d6

nikic reviewed Feb 24, 2015
View changes

Fix return types on error. Avoid a warning on BSD systems.

Loading status checks…

c6fc391

laruence reviewed Feb 25, 2015
View changes

nikic reviewed Apr 9, 2015
View changes

ext/standard/random.c Outdated

RANDOM_G(fd) = fd;

}

ssize_t n = 0;

This comment has been minimized.

Sign in to view

nikic Apr 9, 2015
Member

This needs to be moved into the while look (combined declaration and assignment) to conform with C89.

lt reviewed Apr 9, 2015
View changes

ext/standard/random.c Outdated

Return an arbitrary pseudo-random integer */

PHP_FUNCTION(random_int)

{

zend_long min = ZEND_LONG_MIN;

This comment has been minimized.

Sign in to view

lt Apr 9, 2015
Contributor

@SammyK Since both parameters are required, we don't need these defaults any more.

lt and others added 3 commits Apr 9, 2015

Changes based on feedback

a67e42f

Normalized the return value for errors & updated tests.

f8a6d38

Fix merge conflicts

Loading status checks…

2990341

nikic mentioned this pull request May 9, 2015

Userland CSPRNG #1268

Closed

nikic closed this May 9, 2015

SammyK deleted the SammyK:rand-bytes branch May 11, 2015

This was referenced Jul 7, 2015

Change CSPRNG warnings to exceptions #1397

Closed

Tweak CSPRNG error conditions & add exceptions #1398

Closed

php / php-src

[WIP] Easy User-land CSPRNG #1119

[WIP] Easy User-land CSPRNG #1119

Conversation

SammyK commented Feb 24, 2015

This comment has been minimized.

lt commented on ext/standard/random.c in f8d7aec Feb 14, 2015

This comment has been minimized.

SammyK replied Feb 16, 2015

This comment has been minimized.

SammyK replied Feb 17, 2015

This comment has been minimized.

lt replied Feb 17, 2015

This comment has been minimized.

lt replied Feb 17, 2015

This comment has been minimized.

SammyK replied Feb 17, 2015

This comment has been minimized.

lt commented on ext/standard/random.c in f8d7aec Feb 14, 2015

This comment has been minimized.

lt commented on ext/standard/random.c in 26e4ed2 Feb 14, 2015

This comment has been minimized.

lt commented on ext/standard/random.c in 26e4ed2 Feb 14, 2015

This comment has been minimized.

SammyK replied Feb 20, 2015

This comment has been minimized.

lt replied Feb 20, 2015

This comment has been minimized.

SammyK replied Feb 20, 2015

This comment has been minimized.

lt commented on ext/standard/random.c in 26e4ed2 Feb 14, 2015

This comment has been minimized.

lt commented on ext/standard/random.c in 26e4ed2 Feb 14, 2015

This comment has been minimized.

lt commented on ext/standard/random.c in 7ef5754 Feb 24, 2015

This comment has been minimized.

SammyK replied Feb 24, 2015

This comment has been minimized.

Rican7 replied Feb 24, 2015

This comment has been minimized.

lt commented on ext/standard/random.c in 7ef5754 Feb 24, 2015

This comment has been minimized.

lt replied Feb 24, 2015

This comment has been minimized.

SammyK replied Feb 24, 2015

This comment has been minimized.

lt commented on ext/standard/random.c in 7ef5754 Feb 24, 2015

This comment has been minimized.

nikic Feb 24, 2015 Member

This comment has been minimized.

SammyK Feb 24, 2015 Author Contributor

This comment has been minimized.

nikic Feb 24, 2015 Member

This comment has been minimized.

nikic Feb 24, 2015 Member

This comment has been minimized.

laruence Feb 25, 2015 Member

This comment has been minimized.

patrickallaert Mar 19, 2015 Contributor

This comment has been minimized.

laruence Mar 19, 2015 Member

This comment has been minimized.

SammyK Mar 20, 2015 Author Contributor

This comment has been minimized.

laruence Feb 25, 2015 Member

lt commented Mar 30, 2015

ircmaxell commented Apr 9, 2015

This comment has been minimized.

nikic Apr 9, 2015 Member

SammyK commented Apr 9, 2015

This comment has been minimized.

lt Apr 9, 2015 Contributor

SammyK commented Apr 10, 2015

This comment has been minimized.

nikic commented on f8a6d38 Apr 29, 2015

nikic commented May 9, 2015

SammyK commented May 11, 2015

CodesInChaos commented Jul 6, 2015

SammyK commented Jul 6, 2015

SammyK commented Jul 6, 2015

lt commented on `ext/standard/random.c` in `f8d7aec` Feb 14, 2015

lt commented on `ext/standard/random.c` in `f8d7aec` Feb 14, 2015

lt commented on `ext/standard/random.c` in `26e4ed2` Feb 14, 2015

lt commented on `ext/standard/random.c` in `26e4ed2` Feb 14, 2015

lt commented on `ext/standard/random.c` in `26e4ed2` Feb 14, 2015

lt commented on `ext/standard/random.c` in `26e4ed2` Feb 14, 2015

lt commented on `ext/standard/random.c` in `7ef5754` Feb 24, 2015

lt commented on `ext/standard/random.c` in `7ef5754` Feb 24, 2015

lt commented on `ext/standard/random.c` in `7ef5754` Feb 24, 2015

nikic Feb 24, 2015
Member

SammyK Feb 24, 2015
Author Contributor

nikic Feb 24, 2015
Member

nikic Feb 24, 2015
Member

laruence Feb 25, 2015
Member

patrickallaert Mar 19, 2015
Contributor

laruence Mar 19, 2015
Member

SammyK Mar 20, 2015
Author Contributor

laruence Feb 25, 2015
Member

nikic Apr 9, 2015
Member

lt Apr 9, 2015
Contributor

nikic commented on `f8a6d38` Apr 29, 2015