Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

send keys in parallel #1266

Merged
merged 3 commits into from Mar 26, 2020
Merged

send keys in parallel #1266

merged 3 commits into from Mar 26, 2020

Conversation

@domenkozar
Copy link
Member

@domenkozar domenkozar commented Mar 26, 2020

Fixes #671

not great not terrible but vewy fast

@domenkozar domenkozar force-pushed the parallel-send-keys branch from e40d3e3 to d7c7fc7 Mar 26, 2020
@domenkozar domenkozar requested a review from grahamc Mar 26, 2020
@domenkozar domenkozar force-pushed the parallel-send-keys branch from d7c7fc7 to 4affcfd Mar 26, 2020
Co-Authored-By: Graham Christensen <graham@grahamc.com>
@domenkozar domenkozar force-pushed the parallel-send-keys branch from 03f9601 to dd8cd13 Mar 26, 2020
@domenkozar domenkozar force-pushed the parallel-send-keys branch from dd8cd13 to 37d00d3 Mar 26, 2020
@domenkozar domenkozar requested a review from grahamc Mar 26, 2020

nixops.parallel.run_tasks(
# by default ssh allows 10 sessions per connection via MaxSessions setting
nr_workers=8,

This comment has been minimized.

@grahamc

grahamc Mar 26, 2020
Member

Some day, somebody will send a PR making the 8 configurable for some weird edge case. Today is not that day :P

This comment has been minimized.

@domenkozar

domenkozar Mar 26, 2020
Author Member

:D

@grahamc grahamc merged commit 10e9b1b into master Mar 26, 2020
6 checks passed
6 checks passed
parsing
Details
build
Details
black
Details
mypy
Details
mypy-ratchet
Details
coverage
Details
@grahamc grahamc deleted the parallel-send-keys branch Mar 26, 2020
@dhess
Copy link
Contributor

@dhess dhess commented Mar 28, 2020

This change appears to be effectively forkbombing my Mac. My network isn't very large, something like 10 machines, but I do have probably 5-10 keys per host.

When I run a nixops deploy, it brings the Mac's networking stack to a standstill. :(

edit This is on macOS 10.15.4, btw.

@grahamc
Copy link
Member

@grahamc grahamc commented Mar 28, 2020

Nice. I'll revert for now, we can / should add a global limit on the number of keys being sent at a time.

grahamc added a commit that referenced this pull request Mar 28, 2020
This reverts commit 10e9b1b.
@domenkozar
Copy link
Member Author

@domenkozar domenkozar commented Mar 28, 2020

Huh, 10 connections with 100 sessions kills networking stack? What kind of Mac is this?

@dhess
Copy link
Contributor

@dhess dhess commented Mar 28, 2020

iMac Pro, 64GiB, 10 cores, so no slouch.

I'm as surprised as you are. But there were so many SSH processes running, my nixops deploy hung and I couldn't even ps aux|wc -l to count them all from another terminal.

@grahamc
Copy link
Member

@grahamc grahamc commented Mar 28, 2020

I've seen similar problems, actually. As far as I could tell it had to do with the thundering herd of connections. I wonder if we could pre-calculate how many keys need to be sent, and allocate a total max or something? I dunno, that gets complicated.

@dhess
Copy link
Contributor

@dhess dhess commented Mar 28, 2020

Just tested head, that's much better. Thanks for the remarkably fast turnaround!

Another possibility is this: I'm using gpg-agent to manage my SSH keys, and my SSH private key is stored on a YubiKey. I was under the impression that NixOps always uses the SSH key it generates for root@host at nixops create time for each host, but every once in awhile, NixOps will prompt me for my card's PIN anyway, for reasons I don't understand. So I wonder if some of the SSH sessions are hitting the YubiKey and it, or gpg-agent, just can't keep up?

@dhess
Copy link
Contributor

@dhess dhess commented Mar 28, 2020

After speaking with @grahamc on IRC about this, it sounds like which SSH key NixOps uses is not necessarily deterministic, and that it may use a personal SSH key? If so, anyone who's using a hardware device to manage their SSH private keys is going to have problems with this implementation, assuming I'm right about what was going on here.

@grahamc grahamc added this to the 2.0 milestone Apr 20, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

3 participants
You can’t perform that action at this time.