add "import -"

[mattn]

This is very experimental feature but ndb search should be possible to used while ndb import.

[jb55]

On Sun, Dec 03, 2023 at 11:33:57PM +0900, Yasuhiro Matsumoto wrote: --- a/nostrdb.c +++ b/nostrdb.c @@ -3410,6 +3410,72 @@ int _ndb_process_events(struct ndb *ndb, const char *ldjson, size_t json_len, in return 1; } +static char *read_line(FILE *fp) +{ + char *line = NULL; + size_t maxlen = 0; + + for(;;) + { + char *sep, *block; + + maxlen += BUFSIZ; + line = realloc(line, maxlen + 1); + if (line == NULL) + break; + block = line + maxlen - BUFSIZ; + if (fgets(block, BUFSIZ + 1, fp) == NULL) + { + if (block == line) + { + free(line); + line = NULL; + } + break; + } + sep = strchr(block, '\n'); + if (sep != NULL) + { + *sep = 0x0; + if (sep != block) + { + if ('\r' == *(--sep)) + *sep = 0x0; + } + break; + } + } + return line; +} + +int ndb_process_events_stream(struct ndb *ndb, FILE* fp) +{ +#if DEBUG + int processed = 0; +#endif + + while (!feof(fp)) { + char *line = read_line(fp); + if (line == NULL) + break; + if (!ndb_process_event(ndb, line, strlen(line))) { + ndb_debug("ndb_process_event failed\n"); + free(line); + return 0; + } + free(line); +#if DEBUG + processed++; +#endif + } + +#if DEBUG + ndb_debug("ndb_process_events: processed %d events\n", processed); +#endif + + return 1; +}

Why not just use "getline"? int ndb_process_events_stream(struct ndb *ndb, FILE* fp) { char *line = NULL; size_t len = 0; ssize_t nread; while ((nread = getline(&line, &len, stdin)) != -1) { if (line == NULL) break; ndb_process_event(ndb, line, strlen(line)); } if (line) free(line); }

[jb55]

On Sun, Dec 03, 2023 at 11:38:41PM +0900, Yasuhiro Matsumoto wrote: --- nostrdb.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nostrdb.c b/nostrdb.c index b8db79d..b14ffac 100644 --- a/nostrdb.c +++ b/nostrdb.c @@ -3470,7 +3470,7 @@ int ndb_process_events_stream(struct ndb *ndb, FILE* fp) } #if DEBUG - ndb_debug("ndb_process_events: processed %d events\n", processed); + ndb_debug("ndb_process_events_stream: processed %d events\n", processed); #endif return 1;

you can just squash this into the first patch

[mattn]

sure.

[mattn]

BTW, Why ndb accept three element items? ["EVENT", "s", {...}]

[jb55]

On Mon, Dec 04, 2023 at 06:12:39PM -0800, mattn wrote: BTW, Why ndb accept three element items? `["EVENT", "s", {...}]`

originally nostrdb was designed as a client for consuming events from relays, so this is the format relays returned results to be ingested into nostrdb. These days it's a bit of a client/relay hybrid, so we should probably just have a bare note import, or at least an ingester that supports events of all kinds: ["EVENT", {}] ["EVENT", "subid", {}] {} It's something I will improve in the future.