[PAY-722] Check if audio_tx_hist backfilling complete #4276
Conversation
| "index_user_bank_backfill.py | Tried to check if complete, but no stop_sig" | ||
| ) | ||
| return False | ||
| else: |
There was a problem hiding this comment.
nit: since the if returns, no need for else
There was a problem hiding this comment.
i see that you're doing this for the other indexers too... consider updating those too if you want.
also, it seems that the check_if_backfilling_complete all have similar logic. maybe worth considering if can consolidate
There was a problem hiding this comment.
they do but gonna leave that for now.
discovery-provider/src/app.py
Outdated
| redis_inst.delete("index_user_bank_backfill_complete") | ||
| redis_inst.delete("index_rewards_manager_backfill_complete") | ||
| redis_inst.delete("index_spl_token_backfill_complete") |
There was a problem hiding this comment.
Don't think we need to / want to delete these here - don't we want these bools to persist across container restarts?
There was a problem hiding this comment.
Hmm... not sure it matters as they will be persisted in the db anyways.
| try: | ||
| redis_result = redis.get(index_spl_token_backfill_complete) | ||
| if redis_result: | ||
| is_spl_token_backfill_complete = str(redis_result.decode()) |
There was a problem hiding this comment.
Correct me if I'm wrong but I don't think we can store bools in redis, so I'm storing a string that says either 'false' or 'true' and just checking against those values.
| if redis_complete: | ||
| redis_complete = str(redis_complete.decode()) | ||
| if redis_complete == "true": | ||
| return True |
There was a problem hiding this comment.
nit: if just checking for the existence of this key in redis, we wouldn't don't need this comparison
| index_rewards_manager_backfill_complete, "true" if complete else "false" | ||
| ) |
There was a problem hiding this comment.
for simplicity, let's just not set the key at all if the backfill isn't complete. also, 1 or 0 probs a bit nicer convention to represent this kind of thing than stringifying 'true'
Description
Adds fields in health check that tell us whether the backfill indexing for
audio_tx_historyis complete by caching the results of each indexer'scheck_if_backfilling_completefns in redis. Also checks the redis cache in thecheck_if_backfilling_completefunctions themselves to return early if we've already determined that indexer is complete.Tests
Observed behavior in remote-dev that is indexing prod.
Monitoring - How will this change be monitored? Are there sufficient logs / alerts?
Lots of error logs if something goes south.