Skip to content

rulegen: cannot start any worker on macOS/Windows (spawn + PicklingError), plus several silent failures - #9

Open
bandrel wants to merge 1 commit into
Hydraze:masterfrom
bandrel:pr/rulegen-fixes
Open

rulegen: cannot start any worker on macOS/Windows (spawn + PicklingError), plus several silent failures#9
bandrel wants to merge 1 commit into
Hydraze:masterfrom
bandrel:pr/rulegen-fixes

Conversation

@bandrel

@bandrel bandrel commented Jul 31, 2026

Copy link
Copy Markdown

The blocking one: rulegen cannot start at all on macOS or Windows

Since Python 3.8, multiprocessing defaults to the spawn start method on
those platforms. Spawn pickles the RuleGen instance to reach each worker, and
the instance holds self.hashcat_rule (a dict of lambdas) plus a native Enchant
handle — neither is picklable. Every Process() call therefore dies before any
analysis runs:

_pickle.PicklingError: Can't pickle local object <function RuleGen.__init__.<locals>.<lambda>
when serializing dict item ':'
when serializing dict item 'hashcat_rule'
when serializing RuleGen state

Reproduced against unmodified fd779b2 on macOS with Python 3.14 and
pyenchant 3.3.0. multiprocessing.get_start_method() returns spawn.

Fix: the rule engine, leet map and preanalysis rules move into
build_rule_engine(), and __getstate__/__setstate__ drop the unpicklable
attributes so the child rebuilds them. A custom --wordlist is recorded so
workers rebuild the same dictionary instead of silently falling back to the
system one. After this change a real run completes end to end under spawn.

Ctrl-C did not end a run

analyze_passwords_file prints "Press Ctrl-C to end execution and generate
statistical analysis", but a single SIGINT did not stop the read loop. Two runs
over the same 30,000-password input each processed all 30000 and printed no
interruption notice. SIGINT is now handled as a request to stop reading, so the
loop exits at the next password and the normal shutdown path runs. The same
input now stops after ~3,000 passwords and shuts down immediately, with
statistics and all four output files written.

Shutdown ordering

while not passwords_queue.empty() treated an empty input queue as "work
finished", but it only means the last password was taken, not analyzed, so
the writers could receive their death pill before the workers finished
enqueuing. Shutdown now stops the analysis workers first, joins them, then
pills and joins the writers, so the tail of the analysis survives and the
output files are closed before being read back.

Death pills are delivered with retries rather than abandoned on the first
queue.Full. passwords_queue is bounded to the thread count, so on an early
exit it is normally full; giving up left workers blocked on an empty get()
and turned shutdown into a wait for the terminate fallback.

Also in this PR

  • Worker crash on words with no rules. sorted(words, key=lambda w: len(w["hashcat_rules"][0])) indexes [0] on a list that
    generate_hashcat_rules returns empty whenever nothing survives the
    max_rule_len filter — easy to hit with a small --maxrulelen. The worker's
    except only catches KeyboardInterrupt/SystemExit, so it died and the
    bounded feeder queue could then block the parent forever.
  • Top 10 words percentages were divided by sum(rules_counter.values()).
  • --maxrules was never readself.max_rules was assigned and ignored.
  • --hashcat never verified anythingverify_hashcat_rules had no
    caller. It is now invoked, and reports clearly when the binary isn't where it
    expects rather than doing nothing.
  • The extract rule was keyed ' instead of x, which overwrote the
    truncate rule defined four lines above and left x missing from the table.
  • Hashcat offsets past Z emitted [, \, ] instead of refusing.
  • The progress line reported the previous segment's start offset rather
    than elapsed time.

Testing

Regression tests covering each item are in
bandrel/pack as tests/test_pack.py (24
tests, python3 -m pytest tests/). I kept them out of this PR to keep the diff
to one file — happy to add them here, or to split any of the above into
separate PRs, whichever you prefer.

Related: iphelix#31 documents the subset of these that also affect the
Python 2 original.

…indows

rulegen cannot start a single worker on macOS or Windows. Since Python 3.8
multiprocessing defaults to the 'spawn' start method there, which pickles
the RuleGen instance to reach each worker, and the instance holds a dict
of lambdas (the hashcat rule engine) plus a native Enchant handle. Every
Process() call dies with PicklingError before any analysis runs. The rule
engine moves into build_rule_engine() and __getstate__/__setstate__ drop
the unpicklable attributes so the child rebuilds them.

Ctrl-C is documented as the way to end a run early and still generate
statistics, but a single SIGINT did not stop the read loop: two runs over
the same 30k-password input processed all 30000 and printed no
interruption notice. SIGINT is now a request to stop reading, so the loop
exits at the next password and the normal shutdown path runs.

Shutdown now stops the analysis workers before the output writers and
joins both, so the tail of the analysis is not lost and the files are
closed before being read back. Death pills are delivered with retries;
passwords_queue is bounded, so on an early exit it is normally full and
giving up on the first Full left workers blocked on an empty get().

Also:
- A source word with no surviving rules crashed the worker on an
  empty-list index (words with no rules are now skipped).
- Top 10 words percentages were divided by the rule total.
- --maxrules was never read; it is now enforced.
- --hashcat never verified anything; it now does, and reports clearly
  when the hashcat binary is not where it expects.
- The extract rule was keyed "'" instead of "x", overwriting the truncate
  rule and leaving "x" undefined.
- Hashcat positions past Z emitted stray punctuation; they now raise.
- The progress line reported the previous segment's start offset rather
  than elapsed time.

Regression tests for all of the above are in
https://github.com/bandrel/pack (tests/test_pack.py).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant