Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

#307: Ingest cache #354

Merged
merged 197 commits into from
Jan 15, 2024
Merged

#307: Ingest cache #354

merged 197 commits into from
Jan 15, 2024

Conversation

neilotoole
Copy link
Owner

  • This PR implements an ingest cache DB for document sources such as CSV or Excel.
  • And a download cache for remote document sources.
  • There's a bunch of new cache-related commands, e.g. sq cache clear.
  • We also introduce progress bars for various commands.
  • There are a bunch of new config options related to the new functionality.
  • There's a new text log format for sq's own logs, that outputs human-friendly logs.

@neilotoole
terminal_windows.go issue, again
0badff8
@neilotoole
Removed temp TestPruneEmptyDirTree test
28e007e
@neilotoole
More sqlite file closing
402e20c
@neilotoole
More sqlite file closing, but it's already working
d2a6840
@neilotoole
sqlite type test cleanup
738751a
@neilotoole
Improved logging
17a6944
@neilotoole
Improve Files.Close
d7e11c8
@neilotoole
Minor test cleanup
0e5ce23
@neilotoole
Grips.OpenIngest now more resilient to corrupt cache
1c3f6a8
@neilotoole
Fixed issues with cmd_add_test.go
a8d3f49
@neilotoole
Test refactoring
e037541
@neilotoole
more test cleanup
1f81723
@neilotoole
Speeding up tests
8d41f23
@neilotoole
Speeding up tests, again
a724750
@neilotoole
time testing
3147011
@neilotoole
time testing
fc23cf0
@neilotoole
Update xlsx datetime test data
a450a8d
@neilotoole
Parallel tests
c78e57d
@neilotoole
Text record writer now shows a progress bar while preparing records
7dcad3a
@neilotoole
Refactored pkg progress; added mem usage; tablew shows progress
9d00e15
@neilotoole
Diff cleanup
d10d038
@neilotoole
Implemented sqlite metadata progress update (awkwardly)
e493536
@neilotoole
postgres metadata progress bar
90521e6
@neilotoole
testh .gitignore
2280437
@neilotoole
Grip.SourceMetadata and TableMetadata now have progress bars
03f54b0
@neilotoole
Update go.mod: mpb
8fb9026
@neilotoole
PR review and cleanup
6f67533
@neilotoole
terminal_windows.go issue, again
32cdfa2
@neilotoole neilotoole merged commit db55986 into master Jan 15, 2024
10 of 11 checks passed
@neilotoole neilotoole deleted the feature/307-ingest-cache branch January 15, 2024 01:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@neilotoole