Feature request: add a progress bar to dolt sql < file.sql that shows the progress of a SQL load

[societal-sandpaper]

When running dolt sql < file.sql, I wish there was any indication of how far along the operation is. A progress bar, or a message alongside each Query OK, 464 rows affected (30.94 sec) that says something along the lines of Done line 12/1432 would be helpful.

Context

The reason I'm asking for this is because I am migrating some database dumps from other MySQL databases into dolt, and it takes a while to import 10 GB of SQL database dumps. I wish there was a rough progress bar so I could estimate how long each import will take.

Implementation

As far as implementation goes, one way to do this would be to spin up a thread that goes through the input file, counts the number of newline characters, and then stores that number as the denominator of the progress bar. As dolt goes through the import, it can keep track of how many lines of the input file it has parsed, and use that as the numerator of the progress bar.

[fulghum]

Hey @societal-sandpaper, thanks for the great feature request; this would definitely improve the import experience when running a batch of commands to import data, such as importing a mysqldump. I believe we already do something similar for dolt table import when you are importing a file of data for a single table and it makes sense to provide similar visibility when batching statements to dolt sql.

We've got an uber-issue that covers better progress reporting more broadly (#4076), so I'll link these together for tracking. I'd like to still leave this one open for now to see if it's something we can get to quickly though (e.g. next couple of weeks). It seems like displaying the extra progress message after the status of each batched command would be fairly straightforward and pretty helpful.

[fulghum]

@jennifersp is on the case! 🎉

[jennifersp]

Hi @societal-sandpaper, thank you for the feature request.
I was not able to find a way to resolve this issue because the file is being redirected into dolt sql shell, so that it cannot be accessed. An alternative approach is that dolt sql could be extended to take a file to run similar to imports with csv and json files (i.e. no shell file redirection) and could then show better progress.

[societal-sandpaper]

That makes sense that reading from stdin isn't quite like reading from a normal file pointer. I hadn't considered that complexity.

Digging into it, it appears that, at least on dolt v0.50.0, dolt sql has a --file CLI arg that allows importing from a specific file. Minimal changes required on that front, potentially.

I'll propose the following changes to the dolt sql command.

1. Add a short form for that CLI arg flag, like -f <file> or -fi <file>. Or better yet, change the dolt sql command to support the following syntax, to make it like dolt table import <file>: dolt sql <file>.
2. Add the progress bar, reading from the file location, as discussed in the original ticket here.

[fulghum]

Thanks @societal-sandpaper. That all sounds very reasonable to me.

@jennifersp – could you add this to your queue for this week? At minimum, we should give the --file param a short alias to -f and add some sort of progress reporting there. I'm open to making the file arg work without passing a flag for it, too, but I don't have a strong opinion there and would suggest we start with the existing file param.

[fulghum]

Thanks again for the helpful feedback @societal-sandpaper. @jennifersp has updated dolt sql --file <filename> so that it prints progress information between statements. This looks like it made it out in last night's Dolt release, too.

Please let us know if you have any more feedback for Dolt. We love hearing from customers how we can improve things!

[societal-sandpaper]

I'm excited to test this! Thanks for implementing this so fast.