[QUESTION] Why not use RDB file recovery when AOF files cannot be found?

[ncghost1]

In the loadDataFromDisk , if aof files not found, it will not continue to attempt to recover data through RDB file. but exit the function and create a new AOF file. Is this design intentional?

/* Function called at startup to load RDB or AOF file in memory. */
void loadDataFromDisk(void) {
    long long start = ustime();
    if (server.aof_state == AOF_ON) {
        int ret = loadAppendOnlyFiles(server.aof_manifest);
        if (ret == AOF_FAILED || ret == AOF_OPEN_ERR)
            exit(1);
        if (ret != AOF_NOT_EXIST)
            serverLog(LL_NOTICE, "DB loaded from append only file: %.3f seconds", (float)(ustime()-start)/1000000);
    } else {
        ...

[oranagra]

I don't think these should really be used together (AOF and RDB), and in fact, now that AOFRW doesn't have the overheads it used to have before multi-part AOF, we do intend to consolidate them, see #9795.

Anyway, in what scenario would you have an AOF-enabled server which doesn't have an AOF file and does have an RDB file?

note that even if we'll change the code you suggested to change above, we'll need to immediately trigger an AOFRW (or now with multi-part AOF, rename that RDB file to be a base file, and start a new incremental file) to write the real-time commands into it.

[ncghost1]

Anyway, in what scenario would you have an AOF-enabled server which doesn't have an AOF file and does have an RDB file?

Perhaps the appendonlydir or aof file was deleted due to some unforeseen circumstances (such as human intervention).At this point, although the RDB file is still present, Redis will not attempt to load RDB, resulting in an empty database. This is a strange phenomenon. Perhaps we can prompt users with a warning?

[oranagra]

what do you mean by "prompt"?
we do write to the console / log file (Creating AOF base file %s on server start, and Creating AOF incr file %s on server start), but i don't think that's really helpful, and we obviously can't do anything interactive.

i think the only options to really improve the case (if we detect an RDB file and a missing AOF file) are:

1. terminate the server with some message
2. load an RDB file implicitly

but looking forward, we might disallow enabling both RDB and AOF at the same time, or maybe we'll re-use the AOF manifest to keep track of snapshots, either way, the scenario where the AOF file is gone missing, and the RDB is present would be impossible / invalid.

and even today, the scenario you described is not very likely, and i could even argue that it's likeliness is similar to someone placing the wrong RDB file in that location, so loading it can also load the wrong data set.
i mean, if human intervention error is the reason for this, then that error can go both ways and it's hard to say which solution would fix it and which one would cause more harm.

[ncghost1]

What I mean is print logs, such as informing users you have enabled both aof and rdb . We recommend that you do not enable both, otherwise only aof will be used for data recovery.

I agree with what you said, and the method you provided also sounds better.

It is indeed almost impossible for such a scenario to occur in reality.I was thinking that if Redis only enabled RDB and later shut down the server to enable AOF, it would seem like booting up would result in an empty database. I have asked other people, and they mistakenly assumed that Redis would use RDB to recover when it couldn't find the AOF file.

[chenyang8094]

I think redis does not use RDB file for data recovery when it cannot find AOF file mainly because the data recovery methods of AOF and RDB are different, and using RDB file for recovery may result in inconsistent data. RDB file is a data persistence method of Redis, which takes a snapshot of the database at a specific point in time. AOF file, on the other hand, logs every write operation.

So, I think that when a user enables AOF (indicating they want a strongly consistent data) but the AOF file is missing, Redis should not use RDB file on its own, as it may likely result in inconsistent data.

[ncghost1]

I think redis does not use RDB file for data recovery when it cannot find AOF file mainly because the data recovery methods of AOF and RDB are different, and using RDB file for recovery may result in inconsistent data. RDB file is a data persistence method of Redis, which takes a snapshot of the database at a specific point in time. AOF file, on the other hand, logs every write operation.

So, I think that when a user enables AOF (indicating they want a strongly consistent data) but the AOF file is missing, Redis should not use RDB file on its own, as it may likely result in inconsistent data.

That makes sense, I agree. Thank you for your answer.

[oranagra]

Yes. But if that's the concern it would be very important not to start up empty and accept traffic. Redis would have needed some mechanism to detect it and fail to start..