Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Slow execution due to low entropy in /dev/random #10604

Open
cyberduck opened this issue Feb 8, 2019 · 8 comments
Open

Slow execution due to low entropy in /dev/random #10604

cyberduck opened this issue Feb 8, 2019 · 8 comments
Labels
cli Command Line Interface enhancement

Comments

@cyberduck
Copy link
Collaborator

@cyberduck cyberduck commented Feb 8, 2019

e3d6251 created the issue

  • OS: Ubuntu 16.04
  • Java: openjdk version "11.0.1" 2018-10-16
  • Cyberduck version: 6.9.1 (29776)

duck -h takes more than 5 minutes to execute. I have the impression it runs against a timeout, but I don't see what resources duck needs to print the help information.

This happens for each duck command.

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Feb 8, 2019

e3d6251 commented

Note: in rare cases the response is immediate

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Feb 22, 2019

2ad25be commented

Replying to [10604 jkvbe]:

It takes about 38 minutes.

$ time duck --help
Usage:duck [options...]


URLs must ...
... Purchase a donation key to support the development of this software.

real	38m37.287s
user	0m2.524s
sys	0m0.564s
$

I confirm behaviour since version 6.9.1.29776.
Current testing with ubuntu 18.04 and version 6.9.3.30061 is still the same

On version 6.8.3.29107 duck CLI was running fine. It is independent from OS version (16.04 or 18.04). No java installed since this is no dependency.

  • OS: Ubuntu 16.04
  • Java: openjdk version "11.0.1" 2018-10-16
  • Cyberduck version: 6.9.1 (29776)

duck -h takes more than 5 minutes to execute. I have the impression it runs against a timeout, but I don't see what resources duck needs to print the help information.

This happens for each duck command.

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Mar 28, 2019

a1dfdd4 commented

Hey, we ran into this today and it appears as though the issues is with the use of /dev/random in certain calls that are being made.




If the system in use is low on entropy then /dev/random will block until there are enough bits to satisfy whatever request is being made.




How to test

On a system w/low entropy (Linux VMs that don't see a lot of CPU I/O are good candidates):

In one terminal or screen session run:

time duck -h


In another run:

lsof |grep duck|grep random

You will see calls to both /dev/urandom and /dev/random



Now on that same system (as root) do the following:

(if necessary "ctrl-c" your previously running duck session)

"mv /dev/random /dev/random.orig"

"cp -a /dev/urandom /dev/random"



Once you've replaced /dev/random with /dev/urandom run:

"time duck -h"

It should return a whole lot quicker.

In our case it went from 11m to 8s.


Don't forget to put your system back the way it was:

"rm -f /dev/random"

"mv /dev/random.orig /dev/random"



You can tell how much entropy is available (in bits) for /dev/random by:

"cat /proc/sys/kernel/random/entropy_avail"



In our test case where "duck -h" took 11m to run, we had about 118 bits of available entropy.



Some Suggestions for the devs:

  1. Only be use /dev/random (or libraries that invoke it) if absolutely necessary; otherwise use /dev/urandom
  2. Provide a run-time option that would allow the user to choose the use /dev/urandom exclusively

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Mar 28, 2019

@dkocher commented

Because of this we actually run with -Djava.security.egd=file:/dev/./urandom. We will have to look into when/what libraries still use /dev/random directly.

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Mar 28, 2019

a1dfdd4 commented

Suggestion for End-Users:

If you need duck to work now, and cannot wait for the devs (sometimes this happens).

You can increase the entropy of your systems' /dev/random by using something like haveged.

This acts as a quite decent source of randomness to feed /dev/random in a very efficient (non-CPU-intensive) way. And works excellently for VM/containers that have no access to more traditional sources of randomness.

There are some other projects and methods but something like haveged should be fairly available and infrastructure independent, so no need to worry about virtio / kvm / qemu ...

Also, a big thank you to the cyberduck team for their support.

-Edit:*
To add running duck -h with haveged running with a low-watermark of 2000 (general recommendations but the debian/ubuntu default of 1024 is probably good enough for this case) caused the execution time to go from the 11m area to 3.5s.

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Mar 28, 2019

@dkocher commented

Replying to [comment:10 frwessling]:

Thanks for sharing this!

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Mar 28, 2019

a1dfdd4 commented

Replying to [comment:11 dkocher]:

Replying to [comment:10 frwessling]:

Thanks for sharing this!

My pleasure, glad to help.

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Jan 29, 2021

@dkocher commented

Now documented.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cli Command Line Interface enhancement
Projects
None yet
Development

No branches or pull requests

2 participants