rulegen: cannot start any worker on macOS/Windows (spawn + PicklingError), plus several silent failures - #9
Open
bandrel wants to merge 1 commit into
Open
Conversation
…indows rulegen cannot start a single worker on macOS or Windows. Since Python 3.8 multiprocessing defaults to the 'spawn' start method there, which pickles the RuleGen instance to reach each worker, and the instance holds a dict of lambdas (the hashcat rule engine) plus a native Enchant handle. Every Process() call dies with PicklingError before any analysis runs. The rule engine moves into build_rule_engine() and __getstate__/__setstate__ drop the unpicklable attributes so the child rebuilds them. Ctrl-C is documented as the way to end a run early and still generate statistics, but a single SIGINT did not stop the read loop: two runs over the same 30k-password input processed all 30000 and printed no interruption notice. SIGINT is now a request to stop reading, so the loop exits at the next password and the normal shutdown path runs. Shutdown now stops the analysis workers before the output writers and joins both, so the tail of the analysis is not lost and the files are closed before being read back. Death pills are delivered with retries; passwords_queue is bounded, so on an early exit it is normally full and giving up on the first Full left workers blocked on an empty get(). Also: - A source word with no surviving rules crashed the worker on an empty-list index (words with no rules are now skipped). - Top 10 words percentages were divided by the rule total. - --maxrules was never read; it is now enforced. - --hashcat never verified anything; it now does, and reports clearly when the hashcat binary is not where it expects. - The extract rule was keyed "'" instead of "x", overwriting the truncate rule and leaving "x" undefined. - Hashcat positions past Z emitted stray punctuation; they now raise. - The progress line reported the previous segment's start offset rather than elapsed time. Regression tests for all of the above are in https://github.com/bandrel/pack (tests/test_pack.py).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The blocking one: rulegen cannot start at all on macOS or Windows
Since Python 3.8,
multiprocessingdefaults to the spawn start method onthose platforms. Spawn pickles the
RuleGeninstance to reach each worker, andthe instance holds
self.hashcat_rule(a dict of lambdas) plus a native Enchanthandle — neither is picklable. Every
Process()call therefore dies before anyanalysis runs:
Reproduced against unmodified
fd779b2on macOS with Python 3.14 andpyenchant 3.3.0.
multiprocessing.get_start_method()returnsspawn.Fix: the rule engine, leet map and preanalysis rules move into
build_rule_engine(), and__getstate__/__setstate__drop the unpicklableattributes so the child rebuilds them. A custom
--wordlistis recorded soworkers rebuild the same dictionary instead of silently falling back to the
system one. After this change a real run completes end to end under spawn.
Ctrl-C did not end a run
analyze_passwords_fileprints "Press Ctrl-C to end execution and generatestatistical analysis", but a single SIGINT did not stop the read loop. Two runs
over the same 30,000-password input each processed all 30000 and printed no
interruption notice. SIGINT is now handled as a request to stop reading, so the
loop exits at the next password and the normal shutdown path runs. The same
input now stops after ~3,000 passwords and shuts down immediately, with
statistics and all four output files written.
Shutdown ordering
while not passwords_queue.empty()treated an empty input queue as "workfinished", but it only means the last password was taken, not analyzed, so
the writers could receive their death pill before the workers finished
enqueuing. Shutdown now stops the analysis workers first, joins them, then
pills and joins the writers, so the tail of the analysis survives and the
output files are closed before being read back.
Death pills are delivered with retries rather than abandoned on the first
queue.Full.passwords_queueis bounded to the thread count, so on an earlyexit it is normally full; giving up left workers blocked on an empty
get()and turned shutdown into a wait for the terminate fallback.
Also in this PR
sorted(words, key=lambda w: len(w["hashcat_rules"][0]))indexes[0]on a list thatgenerate_hashcat_rulesreturns empty whenever nothing survives themax_rule_lenfilter — easy to hit with a small--maxrulelen. The worker'sexceptonly catchesKeyboardInterrupt/SystemExit, so it died and thebounded feeder queue could then block the parent forever.
sum(rules_counter.values()).--maxruleswas never read —self.max_ruleswas assigned and ignored.--hashcatnever verified anything —verify_hashcat_ruleshad nocaller. It is now invoked, and reports clearly when the binary isn't where it
expects rather than doing nothing.
'instead ofx, which overwrote thetruncate rule defined four lines above and left
xmissing from the table.Zemitted[,\,]instead of refusing.than elapsed time.
Testing
Regression tests covering each item are in
bandrel/pack as
tests/test_pack.py(24tests,
python3 -m pytest tests/). I kept them out of this PR to keep the diffto one file — happy to add them here, or to split any of the above into
separate PRs, whichever you prefer.
Related: iphelix#31 documents the subset of these that also affect the
Python 2 original.