Extend integration tests

[OmarElawady]

all x+y setup means data namespaces, meta namespaces are always 4 (2+2)

1. recovery test:

• start 4+4 setup
• store a 100MB file
• kill zstor gracefully by sending a sigterm (to make sure all shards are stored)
• delete 4 remote zdb data namespaces
• start zstor again
• check the file is retrievable from the file system with the same checksum

2. recovery failure test:

• start 4+4 setup
• store a 100MB file
• kill zstor gracefully
• delete 5 remote zdb data namespaces
• start zstor again
• check that reading the file returns an error (not malformed data)

3. metadata recovery (it's always 2+2):

• start 4+4 setup and
• store a 100MB file
• kill zstor gracefully
• delete 2 remote zdb meta namespaces
• start zstor again
• check the data is retrievable from the file system with the same checksum

4. test data repair:

• start 4+4 setup but with 12 zdbs in the backend
• store a 100MB file
• kill zstor gracefully
• delete 4 remote zdb namespaces
• start zstor again
• check the data is retrievable from the file system with the same checksum
• wait for 10 minutes + time to upload the data (make the 10 minutes configurable?)
• delete another four
• check the data is still retrievable.

5. test backend config update:

• start 4+4 setup
• store a 100MB file
• kill zstor gracefully
• delete 4 remote zdb namespaces and add new 4
• start zstor again with the same config as the beginning
• replace the config with the old working 4 + the new 4
• send a SIGUSR1 to zstor
• wait for 10 minutes + time to upload the data (make the 10 minutes configurable?)
• delete four namespaces
• check the data is still retrievable.

6. test backend config update for metadata:

• start 4+4 setup
• store a 100MB file
• kill zstor gracefully
• delete 2 remote zdb meta namespaces and add new 2
• start zstor again with the same config as the beginning
• replace the meta config with the old working 2 + the new 2
• send a SIGUSR1 to zstor
• wait for 10 minutes + time to upload the data (make the 10 minutes configurable?)
• delete 2 meta namespaces
• check the data is still retrievable.

7. storage consumed:

• start 4+4 setup
• store 1GB of data
• ensure no more than 2.1GB of data is stored on the remote namespaces

[robvanmieghem]

shouldn't this be in the threefoldtech/quantum-storage repo?

[LeeSmet]

Since these are related to qsfs and not 0-stor specifically, this is better suited as part of threefoldtech/quantum-storage#28