hidane 0.5.1

hidane 0.5.1 fixes a slowdown in 0.5.0: commits with many writes were up to twelve times slower than in 0.4.0. Upgrade if you write in batches. Nothing else changed; parity is as in 0.5.0, verified against the official emulator v1.22.0 and firebase-tools 15.33.0.

Commits (#279)

In 0.5.0, a commit's write results were settled by going over every write once for each document the commit touched (#273, for the results of several writes to one document). The canonical benchmark of 0.5.0 showed it: on the same machine as 0.4.0's (result), a commit of 500 writes took 99.5 ms instead of 8.01 ms, and 100,000 documents in commits of 500 took 19.12 s instead of 1.46 s, slower than the official emulator's 15.96 s. 0.5.1 settles each document once and goes over the writes once; the answers are the same.

The targets check on GitHub-hosted runners now measures a commit of 500 writes too and fails above 25 ms (the official emulator takes 69.2 ms on the canonical machine), so a regression like this one fails on the next push to main.

Breaking changes

None.

Install

curl -fsSL https://hidane.dev/install.sh | sh
brew install hidane-dev/tap/hidane
npm install --save-dev hidane
cargo binstall hidane   # or cargo install hidane
dart pub global activate hidane
docker run --rm -p 8080:8080 ghcr.io/hidane-dev/hidane:0.5.1

Full changelog: v0.5.0...v0.5.1

hidane 0.5.0

Use 0.5.1 instead. Commits with many writes are slow in this release: a commit of 500 writes takes about twelve times as long as in 0.4.0 (#279).

hidane 0.5.0 adds the tools around the emulator: it checks a Firebase project, moves it off the official emulator, measures what that saves, and collects what a bug report needs. It also runs under Testcontainers in place of the Cloud SDK image, and fixes the first two batches of differences that differential fuzzing found against the official emulator.

Verified against the official Cloud Firestore emulator v1.22.0, as firebase-tools 15.33.0 downloads it. Per-client versions are in docs/sdk-certification.md.

Highlights

• hidane doctor checks a Firebase project before it moves to hidane, without running or changing anything:

  • firebase.json and the emulators it starts
  • the Security Rules
  • the SDK and firebase-tools versions, each marked compatible, likely-compatible or not verified
  • the CI workflows and scripts that start the emulators

• hidane migrate shows the changes as a diff and makes them with --apply:

  • hidane exec -- in front of firebase emulators:exec
  • hidane-dev/setup-hidane where actions/setup-java only served the Firestore emulator

  It refuses to write over uncommitted changes and keeps backups.

• hidane benchmark measures your own test command against both emulators on your machine (--compare-official). hidane benchmark savings turns the medians into CI time saved per month, and into cost at a rate you give. It also prints a Markdown case-study block. Figures are medians of your own measurement, with no "N× faster".

• hidane report prints a redacted diagnostic bundle to attach to a bug report:

  • versions, OS, the project scan
  • which official emulator versions are cached
  • a sanitized firebase.json

  Project IDs, keys, tokens, emails and paths are removed. It writes locally and sends nothing.

• Testcontainers: hidane's image works in Testcontainers' Firestore modules for Java and Node by changing only the image name (it reads the gcloud … emulators firestore start command they pass). It is verified with a generic container for Python and Go. See docs/testcontainers.md.

New CLI

Command	Output
hidane doctor [--path DIR] [--format text|json]	hidane.doctor/1
hidane migrate [--path DIR] [--apply [--force]] [--format text|json|patch]	hidane.migrate/1
hidane benchmark [--compare-official] … [-- <command>], hidane benchmark render, hidane benchmark workload	result schema 1
hidane benchmark savings <result.json|-> --runs-per-month N [--cost-per-minute X --currency C]	bench/schema/savings.schema.json
hidane report [--path DIR] [--format markdown|text|json] [--output FILE]	hidane.report/1

The JSON schemas and exit codes of these commands are stable surfaces from this release on (docs/versioning.md).

Compatibility

These answer as the official emulator does. Each is recorded from it and replayed by the tests (#262):

• :batchGet with the same document named more than once returns it once, at its first position.
• Several writes to one document in a commit each report the commit's outcome for that document. For example, there is no updateTime when the last write deletes it.
• A PATCH or POST with an invalid read mask (mask.fieldPaths) writes nothing. Reserved and empty field paths are refused.
• A transform on a field and on a field nested in it, in one write, is refused.
• A RunQuery or RunAggregationQuery without a structured query, a filter of no kind, and sum / avg without a collection are refused. Array-contains in separate branches of an or is allowed.
• Query checks run in the official emulator's order: filters, orders, limit, collection, key filters, cursors and the query vector.

New machine-readable records of what was verified:

• docs/sdk-certification.json
• docs/compatibility.json (the hidane × firebase-tools × official emulator matrix)
• docs/parity-exceptions.json (the intentional differences, each with its kind)

Benchmark

• The canonical benchmark runs both emulators side by side on one dedicated AWS c7i.xlarge, nightly and for each release. Results are published with every sample in hidane-dev/benchmark-results (CC BY 4.0).

• In that environment, the v0.4.0 release run measured these medians:

  Measurement	Official emulator	hidane
  Startup to port	941 ms	1.56 ms
  Idle RSS	148 MiB	7.8 MiB
  firebase emulators:exec around 200 tests	6.75 s	1.67 s

• New scenario: writes with a listener attached (listen_writes). In cargo test it is a regression check that batch writes stay flat as documents accumulate.

Migration

• hidane-dev/setup-hidane is on GitHub Marketplace. Its migrate sub-action opens the migration as a pull request when you start it by hand; it arrives with setup-hidane v1.1.0, after this release.
• examples/ are copy-paste projects that CI runs against hidane: firebase-admin (Node), Python, Go, a Firestore-triggered function, @firebase/rules-unit-testing and a GitHub Actions workflow.

Fuzzing and quality

• A differential fuzzer (tools/fuzz/) sends the same generated requests to the official emulator and to hidane, then minimizes each difference it finds. It runs daily.
• The first triage sorted 143 differences, all of which v0.4.0 already had. This release fixes 39 of them on the seed they were found with. The rest are tracked in #262.
• An upstream watcher and canaries check new firebase-tools, official emulator and SDK releases against hidane before they are verified.

Security and supply chain

• Release binaries build reproducibly: the same commit gives the same bytes on Linux and macOS (docs/reproducible-builds.md).
• Each archive has a CycloneDX SBOM, attested like the archive's build provenance. The container image carries provenance and SBOM attestations too (docs/verifying-releases.md).
• SECURITY.md is revised, and OpenSSF Scorecard runs on the repository. Releases are immutable once published.

Breaking changes

None to a stable surface. The compatibility fixes above can change what a test written against hidane's earlier answers sees, as docs/versioning.md describes.

Known limitations

• The macOS binaries are not signed or notarized: a browser download needs xattr -d com.apple.quarantine hidane (docs/verifying-releases.md).
• iOS, Android and the native FlutterFire platforms are not verified yet (#266, #267). Neither are Datastore mode and the Enterprise edition's pipelines.
• 104 differences found by fuzzing are still open on the original seed (#262). Most concern which error comes first when a request has several problems.

hidane 0.4.0

hidane 0.4.0 delivers Firestore events to the Cloud Functions emulator: Firestore-triggered functions, v1 and v2, fire under firebase emulators:start and emulators:exec with the same data as with the official emulator (v1.22.0), recorded from it and replayed by the test suite.

Functions events (#70)

Every commit sends an event to each registered trigger that watches a document it changed, as the official emulator sends them:

• v1 (providers/cloud.firestore/eventTypes/document.*): JSON with the document before and after, one event ID shared by the v1 triggers of a change.
• v2 (google.cloud.firestore.document.v1.*, withAuthContext too): binary CloudEvents with the ce-* headers and a DocumentEventData in protobuf.
• Path patterns and named databases; batches, transactions, :batchWrite, the Write stream, recursive deletes and cleared databases send events, imports and POST /reset do not.
• Delivered in the background, one at a time, in the official emulator's order; a failed delivery, or a missing --functions_emulator, is a WARNING, as there.

End to end, a functions project (firebase-functions 6.6.0) under firebase emulators:exec --only functions,firestore (firebase-tools 15.33.0) gets the same events with the same data under hidane exec as with the official emulator. Four differences are intentional and documented in parity exceptions: ce-source names the project once (the official emulator writes projects/projects/…); a write that changes nothing and the delete of a missing document send no v2 event (as production and the official emulator's own v1 events); POST /reset keeps the registered triggers (the official emulator forgets them, and functions stop firing silently); a v2 trigger registered without a database watches (default) (the official emulator answers every write with a 500).

Commit checks (#202, #203)

• A commit with several problems now gets the official emulator's first error: names over every write, then the commit as a whole (verifies, sequences of writes to one document), then values, then preconditions and sizes; recorded over REST, gRPC, :batchWrite and the Write stream (4,644 answers).
• Writes to another database's documents stay refused with INVALID_ARGUMENT. The official emulator acknowledges them and keeps them where no read in either database finds them; this is documented as an intentional difference.

Also

• The Go (cloud.google.com/go/firestore), Python (google-cloud-firestore) and Java (google-cloud-firestore) server SDKs are verified against both emulators: documents, queries, transactions, listeners (#187).
• The parity table now counts 20,224 answers recorded from the official emulator.

Install

curl -fsSL https://hidane.dev/install.sh | sh
brew install hidane-dev/tap/hidane
npm install --save-dev hidane
cargo binstall hidane   # or cargo install hidane
dart pub global activate hidane
docker run --rm -p 8080:8080 ghcr.io/hidane-dev/hidane:0.4.0

Full changelog: v0.3.0...v0.4.0

hidane 0.3.0

hidane 0.3.0 completes Web support. firebase-js-sdk's own Firestore browser test suite passes on hidane as it does on the official emulator (v1.22.0), Flutter Web was verified in 0.2.4, and a parity table now counts what the test suite compares with the official emulator.

firebase-js-sdk's browser suite (#74)

tools/js-sdk-browser/ runs the SDK's test:browser:emulator integration tests (Vitest in Playwright's Chromium, over WebChannel and REST) at a pinned commit against an emulator. Against the official emulator and against hidane alike, 1,183 tests pass and 739 skip themselves (pipelines, BSON types, cases marked as not for the emulator); none fail. A new workflow runs the suite against hidane on every change to the server and weekly, and fails on a test that passes on the official emulator but not on hidane.

The suite is not green on either emulator unchanged: some tests leave Firestore instances running, each holding HTTP/1.1 connections for its Listen and Write streams, and Chromium opens at most six connections to one host and port. The runner adds a setup file that terminates the instances each test leaves; the tests themselves are unchanged.

Two differences it found are fixed, each recorded from the official emulator first and replayed by the test suite:

• A commit that writes one document more than once is refused before any write is applied, as the official emulator refuses it, with its INVALID_ARGUMENT messages: an insert (exists: false) after anything but a delete, an update after a delete (Cannot delete then update an entity in the same request.), a delete with exists: true after a delete, and an update_time precondition that differs from the first write's (tools/oracle/same_document_writes.py, 131 cases).
• An equality on __name__ next to an inequality on another field is refused unless __name__ has an inequality too (Equality on key is not allowed if there are other inequality fields and key does not appear in inequalities.), checked in the official emulator's order; past 60 disjunctions a query gets the official emulator's other message (Too many disjunctions after normalization, the maximum is 30).

Parity table (#43)

tools/parity/table.py reads every recording the test suite replays and counts, per area and transport, the official answers its test compares with hidane's, and how: in full, as a Security Rules decision and its reasons, in part, or on the status. It also compares each pair of SDK transcripts. The result is docs/parity.md, the table on hidane.dev and the README badge; CI fails when they fall behind the recordings. Today: 15,427 recorded answers in 39 recordings, 7,328 of them compared in full, 3,912 as a decision and its reasons, 731 in part and 3,401 on the status; 20 pairs of SDK transcripts, 13 identical and 7 with a documented difference; 60 intentional differences in parity exceptions.

Install

curl -fsSL https://hidane.dev/install.sh | sh
brew install hidane-dev/tap/hidane
npm install --save-dev hidane
cargo binstall hidane   # or cargo install hidane
dart pub global activate hidane
docker run --rm -p 8080:8080 ghcr.io/hidane-dev/hidane:0.3.0

Full changelog: v0.2.4...v0.3.0

hidane 0.2.4

hidane 0.2.4 closes the last Security Rules follow-ups of the 0.2 line and fixes how JSON strings are written. Both were recorded from the official emulator (v1.22.0) and are replayed by the test suite.

List rules, coverage and debug() (#167)

A list's filter becomes the constraint the official emulator shows in coverage reports and debug() output, null and NaN included: is not null is NEQ null, a field filter that compares with null or NaN (!= null, == NaN, in [null]) is EQ true, EQ false, which no value satisfies, array-contains null is LIST_CONTAINS [], a repeated filter counts once and a repeated way is evaluated once. The implications follow the official emulator too (bools as false < true, NaN as in IEEE 754, type errors for ordering bools and nulls), and an error inside a parenthesized object is placed where the official emulator places it.

Decisions changed only for the new cases: some queries that return no document are now allowed, and some lists that relied on treating == null or in [null] as a value are now denied, as on the official emulator. One case, is nan next to another filter of the same field, lets a list through that a get of the same documents would deny; hidane follows the official emulator, and the case is open for discussion in #194.

JSON strings (#191)

REST answers escape ", \, control characters, U+2028 and U+2029 as \u with lower-case hex, as protobuf-java's JsonFormat does; error messages also escape / and U+007F–U+009F and U+2000–U+20FF with upper-case hex, as the official error writer does; WebChannel messages also escape < > & = ' in strings and keys, and write doubles and empty maps as the official emulator does, so frame lengths match too.

Install

curl -fsSL https://hidane.dev/install.sh | sh
brew install hidane-dev/tap/hidane
npm install --save-dev hidane
cargo binstall hidane   # or cargo install hidane
dart pub global activate hidane
docker run --rm -p 8080:8080 ghcr.io/hidane-dev/hidane:0.2.4

Full changelog: v0.2.3...v0.2.4

hidane 0.2.3

hidane 0.2.3 fixes firebase emulators:start for projects with Firestore-triggered functions, and verifies @firebase/rules-unit-testing end to end. Both were checked against the official emulator (v1.22.0).

Functions emulator: trigger registrations (#185)

When the Functions emulator loads a function with a Firestore trigger, it registers the trigger with the Firestore emulator (v1: PUT /emulator/v1/projects/{p}/triggers/{key}; v2: POST /emulator/v1/projects/{p}/eventarcTrigger?eventarcTriggerId={key}). hidane answered both with 404, and firebase-tools stopped the whole suite. hidane now answers them as the official emulator does (171 recorded cases: the accepted bodies, 400 for bodies that do not parse, 500 for names and triggers it refuses, and the order of the checks), so the functions load and emulators:start comes up.

Events are not delivered to the Functions emulator yet: hidane keeps the registrations and says so once in firestore-debug.log. Delivery is #70, planned for v0.4.

@firebase/rules-unit-testing under Security Rules (#186)

assertSucceeds / assertFails for signed-in, signed-out and rules-disabled contexts (gets, lists, listeners, writes, batches, transactions) and clearFirestore() give the same results on hidane and the official emulator, in Node (gRPC) and in Chromium (WebChannel), with firebase 13.0.0 and @firebase/rules-unit-testing 6.0.0. Only denial messages differ, where the official emulator lists the reasons of its two evaluation passes (firebase-tools#6252, in docs/parity-exceptions.md). Transcripts are in results/.

Install

curl -fsSL https://hidane.dev/install.sh | sh
brew install hidane-dev/tap/hidane
npm install --save-dev hidane
cargo binstall hidane   # or cargo install hidane
dart pub global activate hidane
docker run --rm -p 8080:8080 ghcr.io/hidane-dev/hidane:0.2.3

Full changelog: v0.2.2...v0.2.3

hidane 0.2.2

hidane 0.2.2 closes two gaps with the official emulator (v1.22.0), each recorded from it and replayed by the test suite.

Security Rules check a commit's verify writes (#173)

The JS and mobile SDKs commit a transaction with a verify write for every document they read but did not write. The official emulator checks each one against the rules as a get of its document (request.inTransaction is true), and so does hidane now: a transaction that read a document the rules do not let it read is denied at the commit, with false for 'get' @ L….

As on the official emulator:

• The writes of a commit are checked in order, and the first denied one denies the commit.
• The rules come before preconditions, for the whole commit: a failing exists precondition next to a denied write answers 403, not 404.
• A verify shares the commit's lookup limits (10 per write, 20 in all). Its document is free to getAfter() and existsAfter() but not to get() and exists().
• A denied commit ends its transaction, over REST, gRPC and the Write stream.

Three differences remain, from the official emulator's two-pass evaluation and from a lookup count that varied between projects. They are listed in docs/parity-exceptions.md.

-0.0 (#174)

The official emulator reads JSON numbers through BigDecimal, which has no negative zero. So -0.0 written over REST or WebChannel becomes 0.0, while -0.0 written over gRPC is stored, read back, exported and imported as -0.0. hidane now does the same.

Writing -0.0 over 0.0 counts as a change, and NaN over NaN does not, as on the official emulator. Exports keep -0.0, and REST prints it as -0.0.

Install

curl -fsSL https://hidane.dev/install.sh | sh
brew install hidane-dev/tap/hidane
npm install --save-dev hidane
cargo binstall hidane   # or cargo install hidane
dart pub global activate hidane
docker run --rm -p 8080:8080 ghcr.io/hidane-dev/hidane:0.2.2

Full changelog: v0.2.1...v0.2.2

hidane 0.2.1

hidane 0.2.1 feeds the Emulator UI's request monitor and gives rules every field of the official request. Both were recorded from the official emulator (v1.22.0) and are replayed by the test suite.

Emulator UI request monitor (#69)

With --websocket_port (which firebase emulators:start passes when the UI runs), hidane serves the official ws://<host>:<port>/requests feed: the last 50 evaluations on connect, then one record per evaluation, in the official format and key order. The Requests tab lists allows, denials and errors as they happen, and its detail view marks the rule lines that decided and shows request.auth, the method, path, resource and time.

Unlike the official emulator, a client that does not read the feed never holds up requests: one that falls more than 16 MiB behind is disconnected, and the UI reconnects for the last 50. The official emulator stalls requests with rules behind such a client; this and the other differences are in docs/parity-exceptions.md.

request as the official emulator gives it (#166)

request.inTransaction, fields, writeFields, readFields, transforms and autoId, and request.query's kind, parent / ancestor, allDescendants, distinct, selectOnlyKeys, groupBy and orderBy (as the query wrote it), with each key in the official order. debug(request) and the coverage report now show the same request as the official emulator, apart from request.time.

Also

• Follow-ups filed: verify writes in a commit are not checked against the rules yet (#173); -0.0 is stored as written, where the official emulator stores 0.0 (#174).

Install

curl -fsSL https://hidane.dev/install.sh | sh
brew install hidane-dev/tap/hidane
npm install --save-dev hidane
cargo binstall hidane   # or cargo install hidane
dart pub global activate hidane
docker run --rm -p 8080:8080 ghcr.io/hidane-dev/hidane:0.2.1

Full changelog: v0.2.0...v0.2.1

hidane 0.2.0

hidane 0.2.0 enforces Security Rules. v0.1 allowed every request; from this release, firestore.rules decides reads, writes, listeners and queries as the official emulator (v1.22.0) does, and each behaviour below was recorded from it and is replayed by the test suite.

Security Rules

• Compiler: rules versions 1 and 2, function and let, the official error messages and positions, and the type checker's warnings (#59, #60). PUT and GET :securityRules work as they do on the official emulator, so firebase emulators:start, @firebase/rules-unit-testing and the Admin endpoints load rules unchanged (#155).
• Evaluation: the 14 types and their methods, request and resource (including request.resource.data after field transforms), get() / exists() / getAfter() / existsAfter() with the 10 and 20 call limits, and mock tokens (alg: none) and Bearer owner (#62–#65, #68).
• Queries: a list or a collection-group query is allowed only when its filter makes every document it may return satisfy the rule, decided with the official emulator's model, constraint values included (#66).
• Denials: the official PERMISSION_DENIED messages, false for 'get' @ L5, evaluation error at L5:22 … and the rest, over gRPC and REST, the Listen and Write streams included.
• Diagnostics: :ruleCoverage as JSON and as the HTML report, and debug() output on stdout (#67).

hidane evaluates each condition once. The official emulator evaluates twice and lists the reasons of both passes in a denial (firebase-tools#6252); this and the other intentional differences are in docs/parity-exceptions.md.

Also

• Documents keep and return their fields in the official emulator's order (#149).
• npm packages are staged and approved with 2FA instead of published directly (#144); every install channel is checked after each release (#148, #153).

Not yet

The Emulator UI request monitor (#69), event delivery to the Functions emulator (#70), the request fields hidane still lacks (writeFields, inTransaction, … #166) and the remaining coverage-report differences (#167) follow in v0.2.x.

Install

curl -fsSL https://hidane.dev/install.sh | sh
brew install hidane-dev/tap/hidane
npm install --save-dev hidane
cargo binstall hidane   # or cargo install hidane
dart pub global activate hidane
docker run --rm -p 8080:8080 ghcr.io/hidane-dev/hidane:0.2.0

Each archive has a SHA-256 in sha256sums.txt and a build provenance attestation: gh attestation verify <archive> --repo hidane-dev/hidane.

Full changelog: v0.1.0...v0.2.0

hidane 0.1.0

The first release of hidane (火種, the seed of fire), a Firestore emulator without Java. It is a single binary that speaks the official Cloud Firestore emulator's gRPC, REST and WebChannel protocols and runs under firebase-tools. Each behaviour is checked against the official emulator v1.22.0.

hidane is an independent project and is not affiliated with or endorsed by Google.

Install

curl -fsSL https://hidane.dev/install.sh | sh   # macOS, Linux: into ~/.local/bin, SHA-256 checked
brew install hidane-dev/tap/hidane               # Homebrew (macOS, Linux)
npm install --save-dev hidane                    # npm, every platform
cargo binstall hidane                            # or: cargo install hidane (Rust 1.88+)
dart pub global activate hidane                  # pub.dev launcher
docker run --rm -p 8080:8080 ghcr.io/hidane-dev/hidane

Then run it under firebase-tools, with no Java installed:

hidane exec -- firebase emulators:start --only firestore

Or run it on its own and point your clients at it:

hidane --host 127.0.0.1 --port 8080
export FIRESTORE_EMULATOR_HOST=127.0.0.1:8080

What 0.1.0 does

• The Firestore API, as the official emulator answers it, error messages included:
  • documents, batch writes and the Write stream;
  • queries with every filter, cursors, collection groups and queries over every collection;
  • count, sum and avg;
  • vector search (findNearest);
  • transactions, with the official emulator's locks and timeouts;
  • listeners, which resume from a token with only what changed.
• One port, three transports:
  • gRPC (with reflection and the v1beta1 name);
  • REST, whose JSON matches the official emulator's byte for byte;
  • WebChannel, which the browser SDK uses.
• firebase-tools: hidane exec -- firebase emulators:start or emulators:exec launches hidane in place of the official jar, with no Java needed. The Emulator UI can browse, create and delete documents and clear data.
• Export and import in the official format, readable by either emulator in both directions: firebase emulators:start --import / --export-on-exit, emulators:export, --seed_from_export, POST …:export / :import.
• The emulator's own endpoints: /reset, /shutdown, clearing a database, CORS, and the Authorization header (owner and mock tokens).
• Speed: the port accepts in 5.5 ms, and hidane idles at 7.4 MiB. 100,000 documents written in batches with a listener attached take 4.9 s, flat per batch. These are measurements, not yet side by side with the official emulator's (#50).

These clients were verified against the official emulator:

• firebase-admin / @google-cloud/firestore
• firebase-js-sdk in Node, Lite and the browser
• @firebase/rules-unit-testing, without rules

Go, Python, Java, iOS, Android and Flutter use the same transports but are not verified yet (compatibility).

Not yet

• Security Rules (v0.2): every request is allowed. --rules is accepted with a warning.
• The Emulator UI's request monitor (#69), events for the Functions emulator (#70), Datastore mode (#81) and Enterprise pipelines (#80).

Where hidane deliberately differs from the official emulator (performance bugs and known defects it does not reproduce) is listed in parity exceptions.

Verifying the downloads

Each archive is listed in sha256sums.txt and carries a build provenance attestation:

sha256sum --check --ignore-missing sha256sums.txt
gh attestation verify hidane-0.1.0-<target>.tar.gz --repo hidane-dev/hidane

The macOS binaries are not signed yet. A binary downloaded through a browser needs xattr -d com.apple.quarantine hidane; curl, Homebrew and npm installs are unaffected.

License

MIT OR Apache-2.0. The archives include the third-party notices (THIRD-PARTY-LICENSES.txt, also printed by hidane --licenses).