7.1 with tenanting upgrade to 7.3 crash

[newhook]

I’ve been testing 7.1 → 7.3 upgrades with tenanted data due to a discussion in Tenants & golang - #8 by newhook which lead to Tenant feature metadata changes in 7.2 release.

I've been testing:

foundationdb-clients_7.1.40-1_amd64.deb foundationdb-server_7.1.40-1_amd64.deb

upgrading to:

foundationdb-clients_7.3.36-1_amd64.deb foundationdb-server_7.3.36-1_amd64.deb

What I discovered was if I have tenanted data in the 7.1 database I end up crashing a 7.3 server on startup. I get one of two crashes:

Apr 10 18:25:02 staging-c-matthew-test fdbmonitor[60480]: LogGroup="default" Process="fdbserver.4500": FDBD joined cluster.
Apr 10 18:25:02 staging-c-matthew-test fdbmonitor[60480]: LogGroup="default" Process="fdbserver.4500": Assertion prefix.size() == TenantAPI::PREFIX_SIZE failed @ /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbclient/Tenant.cpp 45:
Apr 10 18:25:02 staging-c-matthew-test fdbmonitor[60480]: LogGroup="default" Process="fdbserver.4500":   addr2line -e fdbserver.debug -p -C -f -i 0x54a310c 0x4c3d11e 0x33f75ac 0x33f7388 0x33f68a4 0x33f5f43 0x33f5673 0x33f4caa 0x33f4683 0x33f3cc9 0x33f35bc 0x335840b 0x34279cb 0x342791d 0x34262c7 0x289b8f8 0x279bad8 0x279b985 0x279b859 0x279c28e 0x278bb08 0x5385828 0x5385663 0x5390c68 0x5390e68 0x5391ce8 0x5392148 0x5500a98 0x32b1a4b 0x7804a5429d90

or when I was running in memory mode, which I guess is occurring in log replay.

Apr 10 18:18:15 staging-c-matthew-test fdbmonitor[59481]: LogGroup="default" Process="fdbserver.4500": Assertion false failed @ /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbserver/storageserver.actor.cpp 10730:
Apr 10 18:18:15 staging-c-matthew-test fdbmonitor[59481]: LogGroup="default" Process="fdbserver.4500":   addr2line -e fdbserver.debug -p -C -f -i 0x54a310c 0x34685ae 0x3466547 0x33d818e 0x33d62fb 0x33d57a9 0x33d546c 0x33d487a 0x33d3918 0x33d34da 0x33d17d4 0x29f4888 0x29f40c6 0x1ce5298 0x1ce4d03 0x29f1f08 0x29f2d78 0x29f292a 0x29eeb98 0x29ee31c 0x1e8cfc8 0x1e8cd0a 0x534de31 0x534d6d3 0x5500a98 0x32b1a4b 0x7db267e29d90

Are these expected? Anyway around them? Any advice?