Skip to content

Commit

Permalink
Browse files Browse the repository at this point in the history
remove dangerously out-of-date Linux/haveged info
  • Loading branch information
azet committed May 6, 2017
1 parent a43b4c7 commit cf7cef7
Show file tree
Hide file tree
Showing 2 changed files with 0 additions and 56 deletions.
55 changes: 0 additions & 55 deletions src/theory/RNGs.tex
Expand Up @@ -58,61 +58,6 @@ \subsection{When random number generators fail}
time when entropy is low keeps this low-entropy situation for hours
leading to predictable session keys~\cite{HDWH12}.

\subsection{Linux}
\label{subsec:RNG-linux}

%\todo{Other architectures, BSD, Windows?}

On Linux there are two devices that return random bytes when read; the
\verb+/dev/random+ can block until sufficient entropy has been collected
while \verb+/dev/urandom+ will not block and return whatever (possibly
insufficient) entropy has been collected so far.

Unfortunately most crypto implementations are using \verb+/dev/urandom+
and can produce predictable random numbers if not enough entropy has
been collected~\cite{HDWH12}.

Linux supports the injection of additional entropy into the entropy pool
via the device \verb+/dev/random+. On the one hand this is used for
keeping entropy across reboots by storing output of /dev/random into a
file before shutdown and re-injecting the contents during the boot
process. On the other hand this can be used for running a secondary
entropy collector to inject entropy into the kernel entropy pool.

On Linux you can check how much entropy is available with the command:
\begin{lstlisting}
$ cat /proc/sys/kernel/random/entropy_avail
\end{lstlisting}

%% specifics for libraries
%% Openssl uses /dev/urandom. See the paper: https://factorable.net/weakkeys12.conference.pdf (section 5.2)
%% What about other libs?
%% What about other OSes?


\subsection{Recommendations}

To avoid situations where a newly deployed server doesn't have enough
entropy it is recommended to generate keys (e.g. for SSL or SSH) on
a system with a sufficient amount of entropy available and transfer the generated keys
to the server. This is especially advisable for small embedded devices
or virtual machines.

For embedded devices and virtual machines deploying additional userspace
software that generates entropy and feeds this to kernel entropy pool
(e.g. by writing to \verb+/dev/random+ on Linux) is recommended. Note
that only a process with root rights can update the entropy counters in the
kernel; non-root or user processes can still feed entropy to the pool but
cannot update the counters~\cite{Wikipedia:/dev/random}.

For Linux the \verb+haveged+
implementation~\cite{HAV13a} based on the HAVEGE~\cite{SS03}
strong random number generator currently looks like the best choice. It
can feed its generated entropy into the kernel entropy pool and recently
has grown a mechanism to monitor the quality of generated random
numbers~\cite{HAV13b}. The memory footprint may be too high for small
embedded devices, though.

For systems where -- during the lifetime of the keys -- it is expected
that low-entropy situations occur, RSA keys should be preferred over DSA
keys: For DSA, if there is ever insufficient entropy at the time keys
Expand Down
1 change: 0 additions & 1 deletion src/tools.tex
Expand Up @@ -47,7 +47,6 @@ \section{RNGs}
%% NOTE: should we merge that with chapter 6.6??
\begin{itemize*}
\item \href{http://www.fourmilab.ch/random/}{ENT} is a pseudo random number generator sequence tester.
\item \href{http://www.issihosts.com/haveged/}{HaveGE} is a tool which increases the Entropy of the Linux random number generator devices. It is based on the HAVEGE algorithm. \url{http://dl.acm.org/citation.cfm?id=945516}
\item \href{http://www.phy.duke.edu/~rgb/General/dieharder.php}{Dieharder} a random number generator testing tool.
\item \href{http://www.cacert.at/random/}{CAcert Random} another random number generator testing service.
\end{itemize*}
Expand Down

22 comments on commit cf7cef7

@fmarier
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have any more information as to why this information is out of date and why the use of haveged is dangerous?

@Harvester57
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe someone could send an email to the maintainers listed on the Havege project ?

https://www.irisa.fr/caps/projects/hipsor/contact.php

@azet
Copy link
Member Author

@azet azet commented on cf7cef7 Jun 19, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fmarier
See: https://lists.cert.at/pipermail/ach/2017-May/002251.html
..and follow-up discussion on the mailing list. Most decisions are discussed on the mailing list, not GitHub per se (we will reference ML discussion at times on GitHub and GitHub issues vice versa on the mailing list).

@Harvester57
Please go ahead and do so. The project seems barely maintained for years.

@Harvester57
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@azet
Done, waiting for an an answer

@Harvester57
Copy link

@Harvester57 Harvester57 commented on cf7cef7 Jul 10, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got an answer from Andre Seznec (credited as one of the main authors : https://www.irisa.fr/caps/projects/hipsor/contact.php)

He replied that, in his opinion, the principles on which HAVEGE and the haveged daemon are built are still valid, and in fact are more efficient today given the microprocessors architectural evolution (more complex architectures and more non-predictable states usable to gather entropy).

He acknowledged that he did not touch the code for +/- 10 years, and I couldn't not reach the listed maintainer. On Debian, the latest maintainer upload was on november 2016.

He also pointed out a security warning : with some VMs, the hardware cycles counter is emulated and deterministic, and thus predictible. He thereforde does not recommend using HAVEGE on those systems.

@aaronkaplan
Copy link
Member

@aaronkaplan aaronkaplan commented on cf7cef7 Jul 10, 2017 via email

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@azet
Copy link
Member Author

@azet azet commented on cf7cef7 Jul 10, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry but how does it's performance and security compare to the recent changes in the Linux Kernel's CSPRNG implementation, which is solid and very fast? I'm not sure why we should recommend using haveged on Linux systems instead. FreeBSD has been fine for a long time until they fucked up a patch regarding AESNI - but that has been fixed too. So I don't see what haveged would provide in comparison to current implementations of OS CSPRNGs?

We've had this discussion on list and I think it makes far more sense to keep it on there.

@aaronkaplan
Copy link
Member

@aaronkaplan aaronkaplan commented on cf7cef7 Jul 10, 2017 via email

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@azet
Copy link
Member Author

@azet azet commented on cf7cef7 Mar 7, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh I forgot to mention I talked about this issue back in 2017 at SHA and hack.lu:

a couple of people came up to me afterwards with their own custom implementations for "entropy" gathering daemons, some silly, some pretty solid, but they're still by far not as good as what the Linux kernel ships per default these days (and has been for quite a while now). Until the interface will be closed to the unwitting public -- writing to the random device still interferes with the operating systems' CSPRNG, which is really not a good idea to do.

I've heard privately of exploits for various RNGd's -- I can't say anything more and don't have my own exploits of them, mostly because this is a topic I don't want to waste tons of time and energy on as very few people actually use them.

@cstrotm
Copy link

@cstrotm cstrotm commented on cf7cef7 Mar 7, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@azet in some environments, the user might be forced to use an older linux kernel (enterprise linux distribution, embedded space)

I have been recommending "haveged" and "rngd" for virtualized environments in the past.

In some VM environments, there is almost no entropy coming from /dev/random. Generating keys in these environments can take a very long time. Using /dev/urandom is sometimes not an option.

What alternatives to havege exist?

Is "rngd" (https://github.com/nhorman/rng-tools) a better choice (it is actively maintained)?

Red Hat has a (from my point of view, but I'm not an crypto expert) a sensible discussion on the topic:
https://developers.redhat.com/blog/2017/10/05/entropy-rhel-based-cloud-instances/

@cstrotm
Copy link

@cstrotm cstrotm commented on cf7cef7 Mar 8, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi,

I made a video this morning showing the positive effect of Haveged on the entropy coming out of /dev/random on an modern Linux System (fedora 29 Kernel 4.19)

https://strotmann.de/~cas/download/78cb4f98bc96e6c10b7294116cc2515f739f6d46204b1c0258b4d58d/Haveged-Linux-4.19.mp4

This machine runs inside a KVM machine, and without Haveged, it has very little entropy. Generating keys will take very long.

With Haveged running, GPG, SSH, DNSSEC or OpenSSL Key-Generation is almost immediately.

I have never seen "hangs" during key generation on a system with Haveged running, but I have seen "hangs" on systems without Haveged.

I maintain a couple of production DNSSEC signing servers, and without Haveged running, there is a danger that the system locks up during a key rollover event.

I understand that /dev/urandom now is a good (or better) choice, however software often is bound to /dev/random and cannot be changed by the user. Also /dev/random is the portable way to get entropy across Unix systems, at least for now.

I'm not defending Haveged, but I see a need for a source of extra entropy on virtual machines (as shown in the video).

If not Haveged, there might be an alternative. Or in absence of an alternative, we (the community) might need to think about adopting Haveged, find a proper maintainer, do an audit.

@azet
Copy link
Member Author

@azet azet commented on cf7cef7 Mar 8, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm sorry but using /dev/random in general is considered highly unsafe. It's blocking and extremely slow, which can and will cause many applications relying on random input to fail. This interface is not only slow it's deprecated and should not be used for cryptographic purposes as (finally!) documented in man 7 random: http://man7.org/linux/man-pages/man7/random.7.html

   Choice of random source
       Unless you are doing long-term key generation (and most likely not
       even then), you probably shouldn't be reading from the /dev/random
       device or employing getrandom(2) with the GRND_RANDOM flag.  Instead,
       either read from the /dev/urandom device or employ getrandom(2)
       without the GRND_RANDOM flag.  The cryptographic algorithms used for
       the urandom source are quite conservative, and so should be
       sufficient for all purposes.

       The disadvantage of GRND_RANDOM and reads from /dev/random is that
       the operation can block for an indefinite period of time.
       Furthermore, dealing with the partially fulfilled requests that can
       occur when using GRND_RANDOM or when reading from /dev/random
       increases code complexity.

just link to /dev/urandom and you'll get pretty sweet performance especially on such a new kernel. You don't need an "extra source of entropy" on modern Linux kernels, that's a myth. And it'll only make your operating systems CSPRNG weaker, not stronger. It's like saying you need an extra driver for your Intel NIC because the one upstream is not performant enough, because Intel doesn't know it's own hardware. Even if that'd be the case, it'd be up to the driver maintainers to fix it where it needs to be fixed.

I also don't get the concern w.r.t. portability: On BSD and OSX random and urandom are the same thing. Solaris behaves similar. And most software needs to be ported for these Operating Systems anyway and will have #ifdef clauses in it for exactly that purpose.

https://sockpuppet.org/blog/2014/02/25/safely-generate-random-numbers/
https://bugs.ruby-lang.org/issues/9569
(...)

@gcs-github
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cstrotm if you really want better /dev/random performance in your VM, and you don't want to use urandom, with KVM, you can create a virtual hardware RNG with the host's /dev/random as backend. Some sample libvirt XML for it:

    <rng model='virtio'>
      <rate bytes='102400' period='1000'/>
      <backend model='random'>/dev/random</backend>
      <alias name='rng0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </rng>

Then in the VM, have rngd pull from your virtual hardware RNG. You're essentially pooling from the host's /dev/random entropy pool by doing that, if it is acceptable for your use case.

@cstrotm
Copy link

@cstrotm cstrotm commented on cf7cef7 Mar 8, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello Aaron,

I get that. But what should be the recommendation for environments where:

a) the user cannot switch the source of entropy in the application, it is hard coded to /dev/random
b) the user is not permitted to change the device links in the Linux system

Do you have a source on the deprecation of /dev/random?

If /dev/urandom is better in all cases, the switch should be done in the kernel or inside the linux distribution.

@cstrotm
Copy link

@cstrotm cstrotm commented on cf7cef7 Mar 8, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gcs-github I know about KVM and virtual hardware RNG.

Unfortunately creating a virtual hardware RNG in many cases, the clients do not have access to the virtual host (or it might not be a KVM host), maybe because that is managed my a different team inside the organization or is from an external provider.

@azet
Copy link
Member Author

@azet azet commented on cf7cef7 Mar 8, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can bind mount them for example. I agree that distros should do that but most haven't caught up with the kernels developments and then there's always endless discussions like here or with many programming languages that still prefer random or silly custom RNGs (I started or contributed to many change requests for languages and it took ages to convince eg. Ruby to switch). The current Linux man page for the random char device and all related LKML discussions are the reference, that man page caused the myth that random is somehow better and it took quite a while until it was changed although the design of the actual random char device was changed long ago and often improved upon since.

BTW Ruby only finally switched to a reasonable solution after the Linux documentation team changed the random(7) man page. Ignoring expert opinion of many crypto engineers and cryptographers in the relevant change request for years (I linked to all that above in an edit to last my post on GitHub). Python always did the right thing. Erlang still doesn't. Hope that helps. :)

@azet
Copy link
Member Author

@azet azet commented on cf7cef7 Mar 8, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In e.g. a VMWare environment you should use/install the vmware-tools in the guest anyway otherwise you'll lose many features of the hypervisor platform, your clock will go out of sync fast and devices won't be hot pluggable. It depends what you use.

The most laughable solution anyone could come up with was canonical and their virtualization layer. They wrote a service called pollinate that was designed to get an initial seed for virtualized guests via HTTP(!), without any proper exception handling. Unsurprisingly no one used that crap for long. Makes it easy for any attacker on the network to change the seed of the OS CSPRNG network-wide. And in the event there's any service outage or problem with it you'll get similar results. Awesome idea!

@ynezz
Copy link

@ynezz ynezz commented on cf7cef7 May 28, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gcs-github I know about KVM and virtual hardware RNG.

Unfortunately creating a virtual hardware RNG in many cases, the clients do not have access to the virtual host (or it might not be a KVM host), maybe because that is managed my a different team inside the organization or is from an external provider.

@cstrotm I would like to recommend jitterentropy-rngd more details could be found in CPU Time Jitter Based Non-Physical True Random Number Generator paper. This RNG is available as a kernel module since 2015.

There's currently an ongoing effort to add size optimized Jitter-NPTRNG based RNG service/daemon into OpenWrt. It improves the entropy situation significantly on embedded devices and VMs as well.

@aaronkaplan
Copy link
Member

@aaronkaplan aaronkaplan commented on cf7cef7 May 28, 2019 via email

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cstrotm
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ynezz great information, I will check this out. Many thanks.

@azet
Copy link
Member Author

@azet azet commented on cf7cef7 May 29, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ynezz That's exactly what the RNG in Linux does already, it mixes jitter into it's pools and has been doing so for ages (for an analysis of the 2.6.+ RNG: https://eprint.iacr.org/2012/251.pdf). You won't gain more security by yanking out other pools and security features (including backtracking protection and a lot of other things that have been added recently over the past few years) and replacing them with a kernel module that only implements a few of these features. The CPU-specific jitter feature has been upstreamed in 4.2: https://marc.info/?l=linux-kernel&m=143496272614455&w=2

If this is to be put as a recommendation into the guide: please test this properly on relevant embedded devices, show comparative results with the current upstream kernel and if it actually turns out that there's an improvement in security (Many embedded devices don't even have a high resolution timer, so this RNG won't work well at all on these platforms) for embedded devices this should be clearly stated and not as a general remedy. I highly doubt this is true for VMs, there've been a lot of improvements on the side of hypervisors (at least VMWare and KVM) and how they interact with the guest OS since this has been a hot topic in research. I've been tracking guest OS entropy in multiple big virtualization environments and haven't seen any issues with stable-branch Linux distros (RHEL7, Debian stable,..).

@azet
Copy link
Member Author

@azet azet commented on cf7cef7 May 29, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A newer in-depth analysis paper of the RNG in Linux up to kernel version 5.1 by the BSI: https://bsi.bund.de/SharedDocs/Downloads/EN/BSI/Publications/Studies/LinuxRNG/LinuxRNG_EN.pdf?__blob=publicationFile&v=13

By the way written by the same person this daemon/module mentioned above is from and that has been working on upstream Linux Kernel RNG improvements together with a few other people (as well as a complete redesign) over the last couple of years.

Please sign in to comment.