Mnesia is unable to merge the schema for tables using external storage backends

[ieQu1]

Describe the bug
mnesia_schema.erl contains the following code:

otp/lib/mnesia/src/mnesia_schema.erl

Line 3667 in f820bad

change_storage_type(N, ram_copies, Cs) ->

This function is called during schema merging. However, it doesn't handle external storage backends created via mnesia:add_backend_type (e.g. https://github.com/aeternity/mnesia_rocksdb), causing the following error:

(<0.2909.0>) call mnesia_schema:change_storage_type('emqx2@127.0.0.1',{ext,rocksdb_copies,mnesia_rocksdb},{cstruct,emqx_ee_schema_registry_protobuf_cache_tab,set,[],[],[],
         [{{rocksdb_copies,mnesia_rocksdb},['emqx1@127.0.0.1']}],
         0,read_write,false,[],[],false,protobuf_cache,
         [fingerprint,module,module_binary],
         [],[],[],
         {{1686330881043734966,-576460752303423474,1},'emqx1@127.0.0.1'},
         {{4,1},{'emqx1@127.0.0.1',{1687,264314,407434}}}}) ({mnesia_schema,
                                                              merge_storage_type,
                                                              5})

To Reproduce

1. Add an external storage backend (e.g. mnesia_rocksdb:register) in a cluster of two nodes (A and B)
2. Create a table with this backend and ensure it has copies on both nodes.
3. Trigger schema merge. We did it by shutting down B, removing a remote table copy on the surviving node A and restarting B, but there could be an easier way.
4. Mnesia on B fails to start with this error.

Expected behavior
Schema is merged.

Affected versions
Probably all OTP versions that support 3rd party backends.

Additional context

[dgud]

I have problems reproducing this with the instructions you gave, can you write a testcase in mnesia.
There is a table type ext_ets and ext_dets default configured that you can use to remove the mnesia_rocksdb
dependency.

[ieQu1]

Hello,

I think we found a reliable way to reproduce it in our own test suite. Porting to the OTP test suite may take time, since I am not familiar with it, but I might attempt it.

The steps are:

• Create a mnesia cluster with 2 nodes N1, N2
• Create a table T with ext backend that has copies on both nodes
• Shut down N2
• Start a new node N3, and create a copy of T there
• Restart N2
• N2 crashes with the case clause in change_storage_type.

[dgud]

Would still like to have a testcase or some code that I can run which reproduces this.

[IngelaAndin]

ping @ieQu1

[axpxp]

@dgud I have a testcase, how to send it to you?

[dgud]

Post it here, or add a gist.
The testcase should be without mnesia_rocksdb I don't want to debug that.

[axpxp]

@dgud I have re-uploaded erlang27 version, please check, thank you
#8045

[Mikaka27]

I don't think I'm seeing the correct problem when running this:

mnesia_bug_stacktrace.txt

Please verify, not familiar with rocksdb at all.
Would it be possible to have a reproduction without mnesia_rocksdb?

[ieQu1]

Hello,

Sorry for no answer, I was head deep in other stuff. This problem is pretty rare (thankfully), so I don't know the precise conditions to trigger it. I pinpointed the function with a missing clause from the stacktrace, and have a preliminary fix, but no reliable way to test it.

Maybe OTP experts can suggest what scenarios can trigger various types of schema merge. Edit: apparently it's right there #7423 (comment)

[Mikaka27]

Hello,

Sorry for no answer, I was head deep in other stuff. This problem is pretty rare (thankfully), so I don't know the precise conditions to trigger it. I pinpointed the function with a missing clause from the stacktrace, and have a preliminary fix, but no reliable way to test it.

Maybe OTP experts can suggest what scenarios can trigger various types of schema merge.

But you mean that there is an error in your reproduction? And that's why I'm seeing this wrong stacktrace?

But otherwise this reproduction should trigger the error? After running a few times perhaps?

[ieQu1]

Your stacktrace looks different. You likely have stumbled on a different issue that looks specific to rocksdb:

{noproc,
                                       {gen_server,call,
                                        [mnesia_rocksdb_admin,
                                         {rdb,{get_ref,t}},
                                         infinity]}}

This doesn't look like a Mnesia process.

In our case, BUP was not involved. It happened after a regular node restart.

[axpxp]

I don't think I'm seeing the correct problem when running this:

mnesia_bug_stacktrace.txt

Please verify, not familiar with rocksdb at all. Would it be possible to have a reproduction without mnesia_rocksdb?

mnesia:add_backend_type(Alias, Module) after,If the table is empty,Everything is fine,because Module:init_backend first call.
mnesia:add_backend_type(Alias, Module) after,If there's data in the table,mnesia:backup("bk.BUP") after mnesia:install_fallback("bk.BUP") after mnesia:start(),There will be bugs,because Module:init_backend not call.

[Mikaka27]

Hello,

I think we found a reliable way to reproduce it in our own test suite. Porting to the OTP test suite may take time, since I am not familiar with it, but I might attempt it.

The steps are:

* Create a mnesia cluster with 2 nodes N1, N2

* Create a table T with ext backend that has copies on both nodes

* Shut down N2

* Start a new node N3, and create a copy of T there

* Restart N2

* N2 crashes with the case clause in `change_storage_type`.

Couple of questions about this process:

• Does N3 have it's table created with or without external backend?
• Does N3 have it's table created before or after joining the cluster?

[zmstone]

Hi @Mikaka27

• N3 has the table created with external backend
• N3 has the table created before joining the cluster

[Mikaka27]

Hi @Mikaka27

* N3 has the table created with external backend

* N3 has the table created before joining the cluster

Hi, I'm having trouble to reproduce this problem.
Could you share which mnesia (and other) functions (and their arguments) you use in each step?

* Create a mnesia cluster with 2 nodes N1, N2
* Create a table T with ext backend that has copies on both nodes
* Shut down N2
* Start a new node N3, and create a copy of T there
* Restart N2
* N2 crashes with the case clause in `change_storage_type`.

And also it would be good to see how you initialize the schema on each node.