reset(seed=int) is accepted and silently not applied — gymnasium contract inert since the 2024 migration

[spoutop]

Severity: silent — no error, no warning, wrong behaviour.

Summary

NLE.reset(seed=int) - the gymnasium standard seeding contract is accepted,
raises nothing and does not seed the environment. NetHack's core and disp RNGs
never receive the value. Users who follow the gymnasium contract believe they
have a seeded environment and do not.

The documented channel, env.seed(core, disp, reseed, lgen), works correctly;
this report is not a claim that NLE is irreproducible.

Reproduction (no model, no network)

import hashlib, json, gymnasium as gym, numpy as np, minihack  # or nle

def digest(obs):
    n = lambda x: ({"v": x.tolist(), "d": str(x.dtype)}
                   if isinstance(x, np.ndarray) else x)
    return hashlib.sha256(json.dumps({k: n(v) for k, v in obs.items()},
                          sort_keys=True, default=str).encode()).hexdigest()[:16]

E = "MiniHack-Room-5x5-v0"

# gymnasium contract channel — three fresh envs, same seed
print([digest(gym.make(E).reset(seed=42)[0]) for _ in range(3)])
# -> three DIFFERENT digests

# NLE's own channel — three fresh envs, same seed
out = []
for _ in range(3):
    e = gym.make(E); e.unwrapped.seed(core=42, disp=42, reseed=False)
    out.append(digest(e.reset()[0]))
print(out)
# -> three IDENTICAL digests

Observed on 5 of 5 environments at seeds 0, 42 and 12345, across separate OS
processes, fresh in-process constructions, and repeated seeded resets:
NetHackScore-v0, NetHackChallenge-v0, MiniHack-Room-5x5-v0,
MiniHack-MazeWalk-9x9-v0, MiniHack-River-v0.

Before you read further: we checked for a prior report

We searched the issue tracker (state=all, most recent 100 issues/PRs) for an
existing report of this. The only seeding-titled item is facebookresearch#358, "Question: How
does seeding work for NLE?", opened 2023-06-14 ten months before the
migration that introduced the parameter and it is an orientation question
answered with a code-search link, not a report of this behaviour.

We found no duplicate, but the check was not exhaustive: older issues were
not enumerated and GitHub's search API rejected our repo-scoped query. If this
is already known to you, please disregard.

Provenance: this arrived with the gymnasium migration

The parameter has not always existed, and the history localises the fix.

	signature	seed reaches NetHack?
before 867c64998	reset(self, wizkit_items=None)	no such parameter
867c64998, 2024-05-13 — "Migrate from OpenAI Gym to Farama Foundation Gymnasium (#8)"	reset(self, seed=None, options=None)	no
HEAD 2319f2989, 2026-06-12	unchanged	no

Before that commit a caller could not pass a seed to reset and had to use
seed(core, disp), which was correct then and is correct now. The migration
added the signature gymnasium requires without wiring it through, and the
parameter has been accepted-and-inert for 842 days.

We mention this because it is the useful part for triage: the change is
localised to one commit, seed(core, disp) was never broken, and no earlier
behaviour needs restoring.

Scope: your own code is fine, and we found no affected consumer

Two things we checked before writing, because "your seeding is broken" would
have been an unfair summary:

• MiniHack's own package uses the correct channel. minihack/base.py
  calls self.seed(seed, seed, reseed=False) when level seeds are set, and the
  test suite uses env.unwrapped.seed(123456, 789012). Nothing in this report
  suggests MiniHack's published results are affected.
• We looked for a downstream consumer harmed by this and did not find one.
  BALROG wraps both NLE and MiniHack and calls env.reset(seed=seed), which
  looks like exposure — but it routes through a GymV21CompatibilityV0 shim
  that translates the seed into gym_env.seed(seed). We also checked that the
  shim's one-argument form is sufficient: seed(core=42) alone reproduces both
  the reset observation and a fixed 60-step trajectory, 4/4.

So the defect is real and mechanically verified, and its practical exposure is
undemonstrated. We are reporting it because a silently-ignored seed is a
latent hazard for anyone who follows the gymnasium contract, not because we can
point to damage.

Mechanism

nle/env/base.py, NLE.reset:

def reset(self, seed=None, options=None):
    ...
    super().reset(seed=seed, options=options)   # gymnasium.Env.reset
                                                #   seeds only self.np_random
    ...
    self.last_observation = self.nethack.reset(new_ttyrec, options=options)
                                                #   seed is NOT forwarded

gymnasium.Env.reset(seed=…) seeds self.np_random, which NLE does not use for
level generation. The core/disp RNGs are set only via NLE.seed(...).

The sharpest case

NetHackChallenge-v0 deliberately forbids reseeding and says so on its own
channel:

RuntimeError: NetHackChallenge doesn't allow seed changes

Through reset(seed=42) it raises nothing and proceeds unseeded. The
environment's own explicit guard is bypassed, without error, by the interface
most users reach for first.

Why it went unnoticed: the suite tests the other channel

Seeding is tested, thoroughly — but only through seed(core, disp):

• nle/tests/test_envs.py:284 — env0.unwrapped.seed(123456, 789012)
• nle/tests/test_envs.py:351 — env.unwrapped.seed(1234, 5678, False, 1)
• nle/tests/test_tiles.py:84 — same

No test anywhere in the package exercises reset(seed=...). The only two
occurrences of that call are in nle/env/base.py itself.

This is not a case of nobody looking. Seeding has been worked on twice since:

date	commit
2025-02-24	646b70acc	Refactor RNG Seeding functions (#51) — did not touch base.py
2025-04-09	eae89fcd9	Seeding for generating levels (#53) — extended seed() itself (added lgen, updated signature, docstring, return tuple, set_initial_seeds call); did not touch reset

So seed() was edited roughly a dozen lines from the inert reset, and the
contract path still was not caught. We read that as a property of the code and
its coverage, not of anyone's carefulness — a path with no test does not
announce itself, even to someone editing adjacent code.

So the suite exercises the library's own idiom and not the standard's, and will
keep passing for as long as the standard's idiom is inert. We mention it
because it changes what the fix is: wiring the seed through closes the bug,
but a test on the contract path is what stops it recurring the next time the
class is refactored.

Suggested fix (maintainer's call)

Any of:

1. Forward the seed: derive core/disp from seed in NLE.reset when it is
   not None.
2. Raise on reset(seed=…) where the seed cannot be honoured — this is what
   NetHackChallenge already does on its own channel, and it converts a silent
   failure into a loud one.
3. Warn once, and document that seed(core, disp) is the seeding interface.

Option 2 alone would resolve the severity, even without option 1.

Related, lower severity: observation buffers are reused across reset

e = gym.make("MiniHack-Room-5x5-v0")
o1 = e.reset(seed=42)[0]
o2 = e.reset(seed=42)[0]
assert o1["glyphs"] is o2["glyphs"]   # True

A caller who retains an observation and compares it after a later reset
compares the array with itself and always finds equality — so the natural way to
test whether reset reproduces an environment returns a false pass. It also
means a replay buffer storing observations without copying stores aliased data.
Returning copies, or documenting the reuse prominently, would help.

Full write-up and artifacts

Found by a preregistered, zero-model determinism audit
(design/SEED_DETERMINISM_PREREG.md, sha256 87f65dfa…); full per-seed,
per-test, per-channel matrices in results/SEED_DETERMINISM_AUDIT_T3_V2.json.
No claim is made that any published result is affected: which channel a given
paper used is not established by that audit.

---

python seedcheck.py NetHackScore-v0 --import nle --seeds 0 42 12345

Happy to open a PR for the one-line fix and a contract-path test if that is useful.

---

The reproduction above is self-contained and needs nothing else. Happy to open a
PR for the one-line fix plus a contract-path test, or to share the full
per-environment determinism matrices if that would be useful.

One caution worth flagging separately: gymnasium.utils.env_checker.check_reset_seed_determinism
returns PASS on NetHackScore-v0. It retains obs_1 across three later
resets before comparing, so with NLE's reused observation buffers
obs_1 is obs_3 and the comparison comes out true regardless. So running the
official checker would not have caught this — that is a gymnasium-side blind
spot rather than an NLE one, and I am reporting it there too.

[spoutop]

Follow-up: I looked at how widely this is actually hit, since my report above couldn't say.

Searching public code for the pattern — a seed passed to an NLE-family environment — and keeping only files that both construct such an env and call reset(seed=), then collapsing vendored copies by content hash (PufferLib's environments/minihack/environment.py appears in 7 forks and is one codebase), then re-checking each surviving repo across all its Python files for the native seed(core, disp) channel:

15 of 23 repositories never reach the working channel. 6 of those 15 also iterate over seeds and aggregate across runs.

Every filter there lowers the count. The per-file view alone would have overstated it — two repos seed correctly in a sibling file and were moved out at that step.

What it costs, measured with a deterministic policy so no variation is the agent's: under reset(seed=), 6 of 6 reruns at a fixed seed produce different trajectories on 3 of 4 environments, and 71–94% of the variance is within-seed variance a user would take to be zero. Under seed(core, disp) it is exactly 0.000000 on all four and every rerun is byte-identical. MiniHack-Room-5x5-v0 shows no effect at all — a fixed room has no level randomisation to lose — which is the control that makes the other three credible.

I want to be precise about what this is and isn't. I identify no incorrect published result. Silent unseeding breaks seed-based reproduction, which is not the same as invalidating a conclusion, and I assessed none of the 15 for that. The environments are fully seedable through your own channel; what fails is the gymnasium contract layered over it.

Also worth saying plainly: I was not the first to notice this. BartekCupial/codehack has had wrappers/minihack_seed_compat.py since 2026-05-27, which calls env.unwrapped.seed(seed, seed, reseed=False) before delegating to reset(seed=). That predates my report by three months. It suggests the behaviour is known to people who work closely with NLE and has been worked around locally rather than reported.