feat: Add simple data import and export

[fredcarle]

Relevant issue(s)

Resolves #1544

Description

This PR adds import and export functionality to the http api and cli. It can export to json to reduce the potential file size. At this stage csv output was not implemented as it would require extensive type casting (everything in csv is a string) on both writing and reading from the csv.

Tasks

• I made sure the code is well commented, particularly hard-to-understand areas.
• I made sure the repository-held documentation is changed accordingly.
• I made sure the pull request title adheres to the conventional commit style (the subset used in the project can be found in tools/configs/chglog/config.yml).
• I made sure to discuss its limitations such as threats to validity, vulnerability to mistake and misuse, robustness to invalidation of assumptions, resource requirements, ...

How has this been tested?

make test and manual testing

Specify the platform(s) on which this was tested:

• (modify the list accordingly)
• MacOS

[codecov]

Codecov Report

Patch coverage: 67.36% and project coverage change: -0.19 ⚠️

Comparison is base (6d896ba) 75.46% compared to head (661654e) 75.27%.

@@             Coverage Diff             @@
##           develop    #1630      +/-   ##
===========================================
- Coverage    75.46%   75.27%   -0.19%     
===========================================
  Files          203      208       +5     
  Lines        21092    21694     +602     
===========================================
+ Hits         15916    16330     +414     
- Misses        4082     4221     +139     
- Partials      1094     1143      +49     

Flag	Coverage Δ
all-tests	75.27% <67.36%> (-0.19%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
api/http/errors.go	100.00% <ø> (ø)
cli/dump.go	11.11% <0.00%> (ø)
cli/peerid.go	46.03% <0.00%> (ø)
cli/ping.go	38.64% <0.00%> (ø)
cli/request.go	40.37% <0.00%> (ø)
cli/schema_add.go	30.51% <0.00%> (ø)
cli/schema_migration_get.go	38.33% <0.00%> (ø)
cli/schema_migration_set.go	54.14% <0.00%> (ø)
db/errors.go	74.74% <17.14%> (-13.01%)	⬇️
cli/errors.go	25.64% <22.22%> (+0.64%)	⬆️
... and 13 more

... and 7 files with indirect coverage changes

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6d896ba...661654e. Read the comment docs.

[AndrewSisley]

I think the documentation at least needs to change, documenting the memory cost of this operation, if the team is happy with the current implementation in the short-term.

[shahzadlone]

Is a nice first step and I think provides quality of life improvements for users.

I do have a slight nitpick for the terminology of backup/recover over import/export but I think that might be just me :P

suggestion: Would be nice to perhaps include a simple cli usage example in the README.md

[fredcarle]

Note: The tests will fail as I didn't update them for the most recent changes. I want to ensure that we agree with the approach before I put time into the tests which I'll do as soon as consensus is reached.

The change moves the handling of the import and export to the db package as it's a db specific action. It can be done programatically or via CLI. The new approach builds the export file document by document and reads the import file document by document. As such, the memory usage should remain quite low. The obvious downside is that we will only support JSON import/export for now but I feel like that is less important than the benefit of the reduction in memory usage.

Let me know what you guys think :)

[AndrewSisley]

Looks good, but I'd like a few more changes before merge please. I'm only part way through review, but thought I'd submit what I have now and let you get cracking.

[shahzadlone]

thought: I like this backing up and restoring analogy better :), but wondering if should be consistent as currently we have files with the following names:

• backup_export.go
• backup_import.go

Maybe instead can be:

• backup.go
• restore.go

[fredcarle]

The way I went about it is: it's the backup feature and it can export and import data. I wanted a category of actions.

[shahzadlone]

Yep I get that, I feel like backup is being used as an umbrella term. IMO commands will be simpler too if we can pick up one concise word for each action. As pointed out below

[shahzadlone]

thought: If the above thought is expanded then we can have the cli be:

defradb client backup path/to/backup.json
defradb client restore path/to/backup.json

Instead of:

defradb client backup export path/to/backup.json
defradb client backup import path/to/backup.json

[fredcarle]

I had

defradb client export path/to/backup.json
defradb client import path/to/backup.json

before and it was suggested that grouping them under a command / api path would be clearer.

[shahzadlone]

Hmm, I do like the one with less words to type. But if there seems to be a consensus reached already then will just follow that (also is a non-blocker thought).