Closes #1058.

* disable iter metrics

* not comment out for non-iters

* remove explicit drop as clippy cried

)

`spawn_module_host` was changed in #904 to ensure that the supplied
program is indeed the database's current program. There are, however,
uses which would call `spawn_module_host`, followed by either
`init_database` or `update_database` on the returned `ModuleHost`.

This would fail because the condition is not met.

While we cannot prevent misuse, this patch documents the various
lifecycle methods, and adjusts the signature of `init_module_host` to be
(somewhat) consistent with `update_module_host`. Namely, it will
optionally return the result of the `init` reducer call.

Defines traits intended to abstract over the kind of persistence a
database utilizes. The only implementation is (host-)local durability in
terms of the new commitlog crate.

The trait definitions may not be considered stable yet, but are in their
tentative form needed for further integration of the new commitlog.

* Swap the location of tags to go before variant data in the BFLATN encoding

* Fix a comment

* Apply suggestions from code review (@gefjon @Centril)

Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
Signed-off-by: james gilles <jameshgilles@gmail.com>

* Implement memcpy consolidation for sums

* Vanquish clippy

---------

Signed-off-by: james gilles <jameshgilles@gmail.com>
Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>

This patch attempts to integrate the new commitlog with the minimum
changes.

Most of the diff comes from deletions of the legacy log and the need to
adjust tests due to the requirement for a tokio runtime when a durable
database is used in tests.

The "meat" of the patch are the `RelationalDB` constructors,
`RelationalDB::commit_tx`, and the replay logic in
`locking_tx_datastore`.

While `DataKey` is gone, there is still some redundant data being passed
around, which will be addressed in the follow-up patch.

`TxData` is the representation of a transaction after it was committed,
and is passed around for evaluation of subscription queries and sending
the result to clients.

With the new commitlog, it can be represented more compactly, such that
copies for writing to the log can be avoided.

Note that this patch stops short of refactoring `DatabaseUpdate`, which
is another representation of the same information as sent to clients.
This means that `ProductValue`s need to be cloned from `TxData`, just as
before.

Disk usage reporting was left unimplemented in previous patches of the
series, as its semantics are slightly different from before.

Namely, inspecting the size of the commitlog now requires to `stat(2)`
the segment files, and is thus fallible.

Also, a size reporting function is only defined for local durability
(i.e. the commitlog). The behaviour when the database is in a follower
state is left unspecified.

…1085)

* Table: skip alignment checks in eq_row_in_page and hash_row_in_page

* Whoops, those comments can stay the same.

Fix a minor bug where completely empty transactions would still be
written to the commitlog. The bug is minor because, once we start
logging inputs, all transactions will be non-empty.

The check is done in relational DB rather than the durability crate,
because in principle empty transactions are permissible, and may be used
in the future (e.g. to confirm a certain offset).

)

* Fix iai-callgrind rustc version mismatch

* Two spaces

* Future-proof

- Arcing `TableSchema`, and this has benefits elsewhere too.
- Arc<[_]>ing the visitor program instructions.

The data behind the Arcs very rarely change,
which is the perfect case for an Arc.

* Slow query log

* Addressing some PR comments

Prerequisite for auto-disconnect after a database crash, requested for
analytics purposes.

* slow query log: only use Instant::now when needed + refactor

* address Joshua's review

Closes #1115.

Previously subscribers were only identified by their Identity.
However the same Identity can be associated to different Addresses.

* durability: Introduce a method to obtain the max tx offset of a history

Useful for reporting replay progress.
Include note that it is somewhat similar to `std::iter::Iterator::size_hint`.

* core: Re-instantiate replay progress reporting

The percentage is calculated as starting from the zero offset, although
that may change in the future.

When a server key rotation is suspected, suggest to make the new
identity the default.

This usually happens during development using an ephemeral instance. If
one follows the instructions, the `default_identity` is not set for the
existing server, which makes the CLI generate a fresh identity every
time.

See also: #333

Traversing the commitlog without also making it available for writing
would still require upfront I/O imposed by the `open` constructor.

Avoid that by introducing free-standing functions which start traversal
right away.

Co-authored-by: John Detter <no-reply@boppygames.gg>

Avoids an unnecessary roundtrip through `Vec<u8>` in private#731.

Co-authored-by: John Detter <no-reply@boppygames.gg>

Co-authored-by: John Detter <no-reply@boppygames.gg>

…rver> (#1131)

Co-authored-by: John Detter <no-reply@boppygames.gg>

* Remove redundant field Column.col_id

* remove redundant fn Header::ty

* - move extract_table_field to only use (#1137)

- remove unused get_index_by_field

- Nix len-prefixing in `Hash for ProductValue`
- Ignore discriminants in `Hash for ArrayValue`

This adds a non-fallible `write_fmt` method to `CodeIndenter<String>` (since we know it should never fail), which allows to use `write!` and `writeln!` without `.unwrap()` everywhere, making code a lot less noisy.

While working on the new C# codegen, I accidentally noticed that those tests were passing even when they clearly should've been failing due to changed output.

After running with `--nocapture`, I found out it's because the tests are silently skipped and reported as successful when `rust_wasm_test.wasm` isn't built.

This further led to finding that `rust_wasm_test.wasm` is never built - the relevant module results in `rust_wasm_test_module.wasm` instead - so these tests have been incorrectly passing for ages.

This PR changes them to actually build the module as part of testing and updates the snapshots to latest master.

* add hash_bsatn + move proptest generators to sats crate

* add eq_bsatn

While looking through the large diffs while splitting out small PRs out of my refactor branch, I noticed that quite a lot of noise is from me working on a formatted code and using primary constructors while the one in master is not.

As such, I'm splitting out just those automated / non-functional changes into a separate PR to make subsequent functional diffs easier to read.

* [bfops/tests-use-explicit-server]: fix?

* [bfops/tests-use-explicit-server]: empty

---------

Co-authored-by: Zeke Foppa <github.com/bfops>

* [bfops/tests-use-explicit-server]: fix?

* [bfops/tests-use-explicit-server]: empty

* [bfops/default-server-inline]: do thing

* [bfops/default-server-inline]: fix

---------

Co-authored-by: Zeke Foppa <github.com/bfops>

* refactor!: Change DEFAULT_HOST_NICKNAME to "testnet"

- from "local"
- Suggested by Tyler

* refactor!: Change more const to default testnet srv

* [bfops/tests-use-explicit-server]: fix?

* [bfops/tests-use-explicit-server]: empty

* [dylan/refactor/set-default-server-to-testnet]: fix?

* [bfops/default-server-inline]: do thing

* [bfops/default-server-inline]: fix

* [dylan/refactor/set-default-server-to-testnet]: fix

---------

Co-authored-by: Zeke Foppa <github.com/bfops>

* Fail CI on smoketest failures

* Fix smoketests

CLI `generate` code is a particularly heavy user of format-based macros, and so it benefits most from inlining format args where possible.

This is done by adding `#![warn(clippy::uninlined_format_args)]` + running `cargo clippy --fix` followed by `cargo fmt`, so shouldn't require manual review.

This is a follow-up to #1142 and, like that PR, is mainly done to make generate's code a bit cleaner and diffs simpler.

* Simplify source generator csproj

Apparently custom scripts are no longer necessary, so removing to make maintenance simpler (noticed this while working on yet another source generator).

* Add smoketest based on Ingvar's comment, + run smoketests on Windows

* Whoops, don't mkdtemp outside of debugging

* Make smoketest sillier

* Finish up a print statement

* Revert "Make smoketest sillier"

This reverts commit 135b05b.

* Resolve versioning issue in a non-silly, professional way.

---------

Co-authored-by: James Gilles <jameshgilles@gmail.com>

Const members shouldn't count as table or type fields in `[SpacetimeDB.Type]`.

* Disallow calling random lifecycle reducers

* Add lifecycle reducer smoketest

Fixes #1170.

Also updates the bucket values for the queue length histogram.
Also removes the max queue length metric, since the histogram should suffice.

Fixes #1173.

Previously we were only recording this metric for scheduled reducers.
We were also recording it before we acquired access to the module instance.
Now we record it for all reducers after we acquire access to the module instance.

This patch also removes max wait time since the histogram should suffice.

* NFC: few more C# shorthand conversions

For some reason these automated refactoring conversions didn't get included in #1149.

* Also remove unused usings

* Restore a using that was erroneously marked as unused

2. Make `RowRef::row_hash` use the above.
3. Make `Table::insert` return a `RowRef`.
4. Use less unsafe because of 1-3.
5. Use `second-stack` to reuse temporary allocations in hashing and serialization.

Reuse the `indented_block` helper more and add new `CsharpAutogen` helper structure for consistent header / footer structure and reduced boilerplate.

* redefine FieldName as (TableId, ColId)

* fix & refactor tests + move MemTable/Header test helpers test code

* impl PartialEq<ProductValue> for RowRef

* Apply Phoebe's suggestions

Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
Signed-off-by: Mazdak Farrokhzad <twingoow@gmail.com>

---------

Signed-off-by: Mazdak Farrokhzad <twingoow@gmail.com>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>

* impl Eq + Hash for RelValue

* Use Hash for RelValue in incr-eval

* naming: spell out pv, rv, and tro

…PVs in the method (#1162)

* impl Eq + Hash for RelValue

* Use Hash for RelValue in incr-eval

* naming: spell out pv, rv, and tro

* fix soundness hole in Table::delete + don't make + drop PVs

* Clarify `Table::delete`'s callback `before`

Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
Signed-off-by: Mazdak Farrokhzad <twingoow@gmail.com>

---------

Signed-off-by: Mazdak Farrokhzad <twingoow@gmail.com>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>

* [bfops/block-pr-chain]: empty

* [bfops/block-pr-chain]: workflow

* [bfops/block-pr-chain]: fix

* [bfops/block-pr-chain]: update

* [bfops/block-pr-chain]: update

---------

Co-authored-by: Zeke Foppa <github.com/bfops>

* perf(JoinInner): use AVs for keys instead of PVs

* Remove stale comment.

Co-authored-by: james gilles <jameshgilles@gmail.com>
Signed-off-by: Mazdak Farrokhzad <twingoow@gmail.com>

---------

Signed-off-by: Mazdak Farrokhzad <twingoow@gmail.com>
Co-authored-by: james gilles <jameshgilles@gmail.com>

* Make `Page` always fully init

Per discussion on the snapshotting proposal,
this PR changes the type of `Page.row_data` to `[u8; _]`,
where previously it was `[MaybeUninit<u8>; _]`.

This turns out to be shockingly easy,
as our serialization codepaths never write padding bytes into a page.
The only place pages ever became `poison` was the initial allocation;
changing this to `alloc_zeroed` causes the `row_data` to always be valid at `[u8; _]`.

The majority of this diff is replacing `MaybeUninit`-specific operators
with their initialized equivalents,
and updating comments and documentation to reflect the new requirements.

This change also revealed a bug in the benchmarks
introduced when we swapped the order of sum tags and payloads
( #1063 ),
where benchmarks used a hardcoded offset for the tag which had not been updated.

* Update blake3

Blake3 only supports running under Miri as of 1.15.1, the latest version.
Prior versions hard-depended on SIMD intrinsics which Miri doesn't support.

* Address Mazdak's review.

Still pending his agreeing with me that `poison` is a better name than `uninit`.

* "Poison" -> "uninit"

Against my best wishes, for consistency with the broader Rust community's poor choices.

* Remove unnecessary `unsafe` blocks

* More unnecessary `unsafe`; remove forgotten SAFETY comments

* Remove C# GetArgsAsObjectArray

This is a weird "duck-typed" API that has one usage in BitCraft, but it's easy to replace with either real reflection or strong-typed interfaces.

* Update csharp.rs

Signed-off-by: Ingvar Stepanyan <me@rreverser.com>

---------

Signed-off-by: Ingvar Stepanyan <me@rreverser.com>

Closes #1168.

)

* cleanup bsatn ser/de roundtrip test

* fmt::Display for FieldExpr: don't use type_of

* make type_of partial and sound

* type_of: address review feedback

…1208)

* refactor From sql ast type

* remove RelOps.:try_fold (dead code)

* C#: rename DbEventArgs to ReducerContext

This is no longer an event type, so remove subclassing and rename to match the Rust API.

* Update ReducerContext argument name

* Bump to rust 1.78

* Fix lints

* `spacetime build` uses `--project-path` param

* Fix smoketests

* Another small fix

* Another fix

---------

Co-authored-by: John Detter <no-reply@boppygames.gg>

…eanup InMem path (#1211)

* remove dead code wrt. Table

* simplify DbProgram: Insert/Update/Delete only use DbTables

* consistency: move CrudExpr::Update logic to own function

* use TxMode::unwrap_mut more

We talked about this for a while, so decided to do this in a separate PR.

With C# records - which are available since C# 9, so covers Unity requirements as well - we can use subclassing and pattern matching to get sum types that look a lot more like Rust tagged enums.

This is a breaking change but IMO worth it for the better API going forward.

The documentation promised to not collect payload values during folds
(i.e. replaying), but the code did so anyway. This patch makes it so
only values required to satisfy the `Visitor` trait are allocated when
folding.

* [bfops/fix-github-action]: fix?

* [bfops/fix-github-action]: empty

---------

Co-authored-by: Zeke Foppa <github.com/bfops>

* [zeke/noa-router-shuffle]: bring over auth.rs changes from #735

* [zeke/noa-router-shuffle]: review

* Create a lockfile when opening config files

In the past, we've had issues where multiple concurrent CLI processes
would race to read and write the CLI config file,
leading to data loss.

We considered using `flock`/`LockFileEx` and blocking until the file became available,
but unfortunately it's not possible to atomically create and lock a nonexistent file,
which we need to do in the case where the configuration doesn't yet exist.

Instead, we opt for a classic lockfile-based scheme:
Before opening a config file `foo.conf`, attempt to exclusively create `foo.lock`,
and panic if the exclusive creation fails.
Once it becomes clear that we will not write the config any more,
i.e. in `Config::drop`,
delete the lockfile, allowing another process to operate.

This means that attempting to run multiple concurrent Spacetime CLI processes
with the same config file is now a hard error.

* Fix CI failures

This commit fixes two CI failures:

- `spacetime start`, and a few other CLI subcommands, do not access their `Config` at all,
  but the CLI constructs it unconditionally in `main`,
  which made it an error to run any CLI command while `spacetime start` was running.
  This is fixed by having subcommands which don't need a `Config`
  drop it before doing anything.
- Contrary to my assumption,
  the test configuration created by `Config::new_with_localhost` does get `drop`ped,
  because the test harness `clone`s is and passes an owned version to the CLI.
  This was causing it to attempt to delete the empty path, which failed.
  This is fixed by having the home configuration be `Option`al,
  and setting it to `None` in tests.

* Clap before config because they suppress destructors

Perform Clap argument parsing as the very first thing in a CLI process,
before locking the config,
because Clap calls `exit` directly on error rather than panicing
(presumably to have more control over error output),
which prevents destructors from running,
leaving stale lockfiles.

* Encapsulate lockfile logic in a type

Also deduplicate logic for finding config file paths.

* Define `create_parent_dir` helper with comments

* Replace `drop` calls with more explicit `Config::release_lock`.

* [bfops/bump-version]:

* [bfops/bump-version]: bump lockfile

---------

Co-authored-by: Zeke Foppa <github.com/bfops>

* [bfops/codeowners]: do thing

* [bfops/codeowners]: empty

---------

Co-authored-by: Zeke Foppa <github.com/bfops>

* Add SDK test for `SELECT * FROM *`

Which doesn't pass, because we broke it.

* Fix select * from *

---------

Co-authored-by: Phoebe Goldman <phoebe@goldman-tribe.org>

Co-authored-by: Zeke Foppa <github.com/bfops>

* [bfops/codeowners]: empty

* [bfops/codeowners]: update CODEOWNERS

---------

Co-authored-by: Zeke Foppa <github.com/bfops>

* [bfops/codeowners]: empty

* [bfops/codeowners]: update CODEOWNERS

* [bfops/codeowners]: update

---------

Co-authored-by: Zeke Foppa <github.com/bfops>