panicked at 'Maximum offset must be present after snapshot'

[DeoLeung]

Describe the problem
readyset was running 2023-01-18,
today upgrade to image 2023-02-15 and change a few settings
add --replication-tables=schema1.*,schema12.*,schema13.*,schema14.*
add --prometheus-metrics

after restart, readyset drop some previous replication (as expected), then crash

To Reproduce
current docker-compose.yml

readyset:
    image: public.ecr.aws/readyset/readyset:beta-2023-02-15
    ports:
      - "5444:5433"
      - "6034:6034"
    environment:
      DEPLOYMENT_ENV: staging
      RUST_BACKTRACE: 1
      RUST_BACKTRACE: full
    command:
      - --standalone
      - --replication-tables=schema1.*,schema12.*,schema13.*,schema14.*
      - --deployment=staging
      - --disable-telemetry
      - --database-type=postgresql
      - --upstream-db-url=postgresql://abc
      - --address=0.0.0.0:5433
      - --username=a
      - --password=b
      - --query-caching=explicit
      - --db-dir=/state
      # metrics
      - --query-log
      - --prometheus-metrics
      - --query-log-ad-hoc

Expected behavior
A clear and concise description of what you expected to happen.

Additional data / screenshots

readyset_1                   | thread 'tokio-runtime-worker' panicked at 'Maximum offset must be present after snapshot', /tmp/readyset/replicators/src/noria_adapter.rs:553:14
readyset_1                   | stack backtrace:
readyset_1                   |    0:     0x55909bf950e0 - <unknown>
readyset_1                   |    1:     0x55909bfbc15c - <unknown>
readyset_1                   |    2:     0x55909bf8e575 - <unknown>
readyset_1                   |    3:     0x55909bf967e1 - <unknown>
readyset_1                   |    4:     0x55909bf964b3 - <unknown>
readyset_1                   |    5:     0x55909bf96ee3 - <unknown>
readyset_1                   |    6:     0x55909bf96dd7 - <unknown>
readyset_1                   |    7:     0x55909bf95604 - <unknown>
readyset_1                   |    8:     0x55909bf96b02 - <unknown>
readyset_1                   |    9:     0x55909bfb94e3 - <unknown>
readyset_1                   |   10:     0x55909bfb93a1 - <unknown>
readyset_1                   |   11:     0x55909bfb934b - <unknown>
readyset_1                   |   12:     0x55909bfb90a6 - <unknown>
readyset_1                   |   13:     0x559099c1bdae - <unknown>
readyset_1                   |   14:     0x559099c658db - <unknown>
readyset_1                   |   15:     0x559099947a98 - <unknown>
readyset_1                   |   16:     0x5590999dfcae - <unknown>
readyset_1                   |   17:     0x55909986aca7 - <unknown>
readyset_1                   |   18:     0x55909be27fff - <unknown>
readyset_1                   |   19:     0x55909be26f83 - <unknown>
readyset_1                   |   20:     0x559099b61c9a - <unknown>
readyset_1                   |   21:     0x5590999e0be8 - <unknown>
readyset_1                   |   22:     0x559099862eb6 - <unknown>
readyset_1                   |   23:     0x55909be17ad1 - <unknown>
readyset_1                   |   24:     0x55909be18237 - <unknown>
readyset_1                   |   25:     0x55909be295cf - <unknown>
readyset_1                   |   26:     0x55909bf9a933 - <unknown>
readyset_1                   |   27:     0x7f0185456609 - start_thread
readyset_1                   |   28:     0x7f0184ea0133 - clone
readyset_1                   |   29:                0x0 - <unknown>

If applicable, add screenshots to help explain your problem.

Environment

• ReadySet version [beta-2023-02-15]
• ReadySet deployment method [docker]
• Upstream database and version [Postgres 13]

Additional context
changing back to old config won't help.

currently I solved it by deleting the data directory /state

[glittershark]

Thanks for the bug report @DeoLeung! As you surmised, this is happening because we don't currently support upgrading to new versions of ReadySet without wiping out the data directory. I'm actually currently working on making that happen automatically - soon, you'll always be able to upgrade to a new version of ReadySet in-place by just updating to a new binary, without having to wipe out any state. For now, all upgrades require wiping out the state directory before running the new binary.

[jseldess]

Sorry you ran into this, @DeoLeung. We have a note at the top of our releases page suggesting the need to wipe data files: https://docs.readyset.io/releases/readyset-core/. But that's not very helpful. I'll make sure we get an upgrade page added to the docs before long.

[glittershark]

ReadySet should now automatically wipe out snapshotted tables and re-snapshot on upgrade.