Skip already ingested items when rerunning ingestion script

[wirthual]

I hit a rate limit when ingesting messages and need to rerun the ingestion script.

Is there a way to skip items that have already been successfully processed and are already present in the storage?

[gvanrossum]

That's a good question. Unfortunately in ingest_vtt.py there is not. In the demo code there is not. In tools/test_email.py (poor name) there is -- that program stores a bit per message in a "shelf". Search for index_log.

In general this is considered outside the scope of the typeagent package, though we may have to reconsider that decision.

[gvanrossum-ms]

Well, I am adding a very primitive mechanism to podcast_ingest(). You can set a verbose flag and a batch_size arg and then it will print which messages were successfully indexed immediately after add_messages_with_indexing() for each batch completes. When it crashes or you ^C it, you can pass the start_message flag to the first messages not yet added. When there's a problem with a particular message, you can set batch_size=1 to pinpoint which message, and set start_message to the next message.

We should add such functionality to ingest_vtt.py (assuming that's what you're using -- the ingest.py demo leaves such stuff out to preserve its "hello world" nature).

Changing podcast_ingest.py is going to be one of my next PRs.

[gvanrossum-ms]

This has now landed in main.

The code is at the end of typeagent/podcasts/podcast_ingest.py (main()).

If you wrote your own ingestion script, you can use a similar technique. The key for restarting is to have stable message numbers ("ordinals") and the ability in your driver to both show message numbers of each batch successfully added, and allow setting the number of messages to skip from the front.

If you're using tools/ingest_vtt.py you'd have [EDIT] to add similar functionality there.

[gvanrossum-ms]

Shall I keep this open until the functionality is added to ingest_vtt.py ?

[wirthual]

Hi,

I checked typeagent/podcasts/podcast_ingest.py and understand now how to work around my the problem. Thank you.

From my side this can be closed.