Importer V3 #395

conradludgate · 2022-05-07T14:35:04Z

The old API was clever, but sacrificed a lot of flexibility.

This is much more naive, simple and should be more flexible and probably more performant too

conradludgate · 2022-05-07T14:36:20Z

src/command/mod.rs

-            Self::Client(client) => client.run().await,
+            Self::Client(client) => client.run(),
            #[cfg(feature = "server")]
-            Self::Server(server) => server.run().await,
+            Self::Server(server) => server.run(),


This is a side change. The client does no tokio::spawn calls, so it doesn't need to be multi-threaded like the server. So this defers the tokio setup to the client/server respectively.

conradludgate · 2022-05-07T14:38:26Z

atuin-client/src/import/mod.rs

-pub trait Importer: IntoIterator<Item = Result<History>> + Sized {
+#[async_trait]
+pub trait Importer: Sized {
    const NAME: &'static str;
-    fn histpath() -> Result<PathBuf>;
-    fn parse(path: impl AsRef<Path>) -> Result<Self>;
+    async fn new() -> io::Result<Self>;
+    async fn entries(&mut self) -> Result<usize>;
+    async fn load(self, loader: &mut impl Loader) -> Result<()>;
+}
+
+#[async_trait]
+pub trait Loader: Sync + Send {
+    async fn push(&mut self, hist: History) -> eyre::Result<()>;
+}


This is the core API change. It's very simple, the importer can

Be created

Count the number of entries it may contain

Load all of it's entries into a Loader. (A TestLoader is defined below for testing)

All these methods are async and fallible so I'm not sure how much more flexible we can be 😅

conradludgate · 2022-05-07T14:39:48Z

src/command/client/import.rs

+        self.pb.inc(1);
+        self.buf.push(hist);
+        if self.buf.len() == self.buf.capacity() {
+            self.db.save_bulk(&self.buf).await?;
+            self.buf.clear();
+        }
+        Ok(())


Much simpler batching. Push, check if at capacity, drain. No more iterator fuckery

ellie

🚀 🚀 🚀

inb4 a year from now "wow this code I wrote a year ago needs fixing" 😂

06ac958 Show current version on server index (#436) 706b1af Disable ARM docker builds (#438) f2abc23 Update README.md 3c2b055 Noyez fix dir hostname utf8 (#430) 3f5350d [feature] Add scroll wheel support to interactive history search (#435) dcdde22 Fix text outline for dark mode 9ac0c60 Implement cursor (#412) 119ab9e Adds password prompt for register and login (#424) e5df809 Noyez zsh histdb import (#393) b08e254 Improve default fish keybindings (#420) 4096c6e Update README.md cd2a3ab Add fish shell to key binding docs (#418) b278211 Bump clap_complete from 3.1.3 to 3.1.4 (#397) ee66c0a Bump axum from 0.5.5 to 0.5.6 (#415) 4297e26 Bump tokio from 1.18.1 to 1.18.2 (#396) dbd9ca5 Bump clap from 3.1.16 to 3.1.18 (#401) a7c9d19 Bump tower-http from 0.3.2 to 0.3.3 (#399) 3b79f68 Bump axum from 0.5.4 to 0.5.5 (#402) f340771 Cleanup dependencies – disable unnecessary or unused features (#407) ab294cd Don't pollute shell environment - remove 'id' variable (#408) 14b3060 Allow to build atuin server without client (#404) 5e4e8d1 Don't create config dir for server in default location if not needed (#406) b7946cc Update Chinese version README.md (#403) e0291f6 Update README.md 301190e Build ARM docker image in GitHub Actions using QEMU (#400) 1d030b9 Importer V3 (#395) d3a4ff9 Bump clap from 3.1.15 to 3.1.16 (#392) e9d2ec4 Add ctrl-k and ctrl-j for up and down (#394) 25afb5b Bump serde_json from 1.0.80 to 1.0.81 (#387) 4a839da Adds stats summary (#384) 7a394b0 Bump serde from 1.0.136 to 1.0.137 (#375) edd3f81 Bump clap_complete from 3.1.2 to 3.1.3 (#377) d85d03d Bump log from 0.4.16 to 0.4.17 (#382) dc3b7ef Bump tokio from 1.18.0 to 1.18.1 (#383) 12440c1 Bump serde_json from 1.0.79 to 1.0.80 (#376) 731042f Bump tower-http from 0.3.1 to 0.3.2 (#378) 82505e6 Bump clap from 3.1.12 to 3.1.15 (#381) e05c19d Add Chinese documentation translation & Fix spelling mistakes (#373) 6e280e2 Add Russian documentation translation (#365) 40efdd1 Bump http from 0.2.6 to 0.2.7 (#368) 8bc5bec Bump tower-http from 0.3.0 to 0.3.1 (#367) 172ac8d Create FUNDING.yml 7cdd00b Bump tokio from 1.17.0 to 1.18.0 (#357) 9d2e9ea Search: Allow specifiying the limited of returned entries (#364) 93ab4e7 ignore JetBrains IDEs, tidy-up imports (#348) 2cb4cb3 Bump axum from 0.5.3 to 0.5.4 (#355) 796644e Add created_at column to users (#354) f8233bc SQLx cannot run this migration OK (#353) d8ef5dd fix db range query (#351) 5926ea6 fix import auto for bash (#352) 43d299f bump tui (#346) 8ac6571 Remove all select * from the server queries (#347) 4030de4 Add btree index on history table (#345) b692e0c Bump tower-http from 0.2.5 to 0.3.0 (#343) 3680f4a Bump clap from 3.1.11 to 3.1.12 (#342) 7f5310a history list (#340)

start of importer refactor

2f5a897

conradludgate commented May 7, 2022

View reviewed changes

fish

4290a1c

conradludgate mentioned this pull request May 7, 2022

Noyez zsh histdb import #393

Merged

conradludgate added 2 commits May 8, 2022 10:29

resh

b8ac4e2

zsh

440ec1f

conradludgate marked this pull request as ready for review May 8, 2022 10:09

ellie approved these changes May 9, 2022

View reviewed changes

ellie merged commit 1d030b9 into main May 9, 2022

ellie deleted the importer-v3 branch May 9, 2022 06:46

norpol mentioned this pull request May 31, 2022

zsh-histdb import #93

Closed

ellie mentioned this pull request Jun 6, 2022

Release v0.10.0 #440

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Importer V3 #395

Importer V3 #395

conradludgate commented May 7, 2022

conradludgate May 7, 2022

conradludgate May 7, 2022

conradludgate May 7, 2022

ellie left a comment

Importer V3 #395

Importer V3 #395

Conversation

conradludgate commented May 7, 2022

conradludgate May 7, 2022

Choose a reason for hiding this comment

conradludgate May 7, 2022

Choose a reason for hiding this comment

conradludgate May 7, 2022

Choose a reason for hiding this comment

ellie left a comment

Choose a reason for hiding this comment