[bazarr] Add bounded concurrency to episodes endpoint #238

rtrox · 2023-10-29T00:49:33Z

Description of the change

This PR adds pagination to collection from the /api/episodes endpoint in the bazarr collector. This pagination uses bounded concurrency, sending no more than --series-batch-concurrency (default 10) batches to the bazarr server at a time, and asking for no more than --series-batch-size (default 50) seriesIds at a time.

fixes #237

A test image is available at ghcr.io/rtrox/exportarr:episodes-concurrency.

Benefits

Allows retrieval of series metrics from bazarr instances with a large number of serieses.

Possible drawbacks

More goroutines?

Applicable issues

fixes Bazarr failures #237

Additional information

phyzical · 2023-10-30T00:54:13Z

@rtrox ill run this later tonight to get some benchmark numbers for comparision

rtrox · 2023-10-30T03:01:48Z

@phyzical - that'd be great, I don't actually have a bazarr instance to test against >.< I'm reasonably confident from the unittests, but mind comparing the metric values too and just double checking that there aren't any logic bugs in the counting?

rtrox · 2023-10-30T03:14:45Z

@phyzical - one more note -- in lidarr & sonarr we have a similar pattern, where a few specific metrics require us to send an API call per object. In both places, we hide these metrics behind the EnableAdditionalMetrics config value. I haven't made that change here yet, as it looks like I'd be gating the majority of the interesting metrics, but I'd like to think through what metrics are actually vital, and see if we can find ways to get them from the higher order APIs (or make some PRs to bazarr to do so).

phyzical · 2023-10-30T03:58:49Z

i did have that if sitting there ready for heavy lifting, but as you've guessed it was kinda of all or nothing.

The other reason i didn't bother was that i thought i had a fairly large sample size (3k+ series) and i was able to get it all within 30 seconds, which was more than enough for my usecase (checking a dashboard once in a blue moon)

i personally think we should maybe be suggesting that these exporters are configured for longer intervals i.e once every 5-10 minutes anyway, that way if it is an expensive scrape it doesn't matter as much. the only outlier i can think of is alerts tied to disk space/health check, but even then id say finding out at 5-10 minutes for hobby stuff is still decent.

But ill give it a test still because if its does speed things up why not!

rtrox · 2023-10-30T05:00:53Z

Yea, that makes sense. The only thing to keep in mind is that grafana's prometheus integration's max_source_resolution is somewhat low (I think it's a minute, but don't quote me on that). So if we decided to give that advice, we'll need to document the process of adding max_source_resolution as a query parameter to the grafana prometheus integration, given most of our users are hobbyists.

That said, breaking this up was probably necessary anyway, in the linked issue the url alone was near 100K, which means the memory needed to deserialize the response was definitely going to be a bit crazy >.<.

phyzical · 2023-10-30T06:06:05Z

yeah i saw the size of the log in the 20 mb range 😆

TBH ive been grafanaing for about 4 months so good to know the grafana limitation.

But i think another part of the issue here is also that we assume each request should perform a new scrape, most of the exporters ive used tend to work with a scrape interval internally and cache the result for the prometheus instance to just get served some cold hard data.

Not saying that should be done in this pr. But something that should be considered if it hasnt already.

(i assume itll be a "oh its on the list" and there just wasnt enough justification)

rtrox · 2023-10-30T07:15:40Z

Believe it or not, caching and scrape intervals are actually anti-patterns in Prometheus exporters 🙃 From the exporter instrumentation docs:

Metrics should only be pulled from the application when Prometheus scrapes them, exporters should not perform scrapes based on their own timers. That is, all scrapes should be synchronous.

Accordingly, you should not set timestamps on the metrics you expose, let Prometheus take care of that. If you think you need timestamps, then you probably need the Pushgateway instead.

If a metric is particularly expensive to retrieve, i.e. takes more than a minute, it is acceptable to cache it. This should be noted in the HELP string.

The default scrape timeout for Prometheus is 10 seconds. If your exporter can be expected to exceed this, you should explicitly call this out in your user documentation.

We have a couple places that we do cache - in prowlarr, for example, as prowlarr exposes stats in an odd way which requires us to calculate and cache deltas to create accurate counts. Otherwise, we generally avoid caching and stick with synchronous calls.

phyzical · 2023-10-30T08:07:33Z

yeah that makes sense. in most cases just adjusting interval and scrape times would suffice (has for me).

And its probably like anything prometheus keep the rules simple and locked down and it should just work out the box.

The caching is it a simple toggleable feature or is it fairly deep? it does feel like an edgecase at this point i would personally consider my library the large end but this other users has double what i do so 😆 (no saying it shouldnt be a feature but should depend on how complicated it becomes to support optional caching)

i did actually try asyncing the calls originally too but found it made things slower and i think i'm seeing the same during testing?

old:

# HELP bazarr_episode_subtitles_downloaded_total Total number of downloaded episode subtitles
# TYPE bazarr_episode_subtitles_downloaded_total gauge
bazarr_episode_subtitles_downloaded_total{url="http://bazarr:6767"} 64927
# HELP bazarr_episode_subtitles_filesize_total Total filesize of all episode subtitles
# TYPE bazarr_episode_subtitles_filesize_total gauge
bazarr_episode_subtitles_filesize_total{url="http://bazarr:6767"} 3.256260812e+09
# HELP bazarr_episode_subtitles_history_total Total number of history episode subtitles
# TYPE bazarr_episode_subtitles_history_total gauge
bazarr_episode_subtitles_history_total{url="http://bazarr:6767"} 17952
# HELP bazarr_episode_subtitles_missing_total Total number of missing episode subtitles
# TYPE bazarr_episode_subtitles_missing_total gauge
bazarr_episode_subtitles_missing_total{url="http://bazarr:6767"} 18619
# HELP bazarr_episode_subtitles_monitored_total Total number of monitored episode subtitles
# TYPE bazarr_episode_subtitles_monitored_total gauge
bazarr_episode_subtitles_monitored_total{url="http://bazarr:6767"} 26877
# HELP bazarr_episode_subtitles_unmonitored_total Total number of unmonitored episode subtitles
# TYPE bazarr_episode_subtitles_unmonitored_total gauge
bazarr_episode_subtitles_unmonitored_total{url="http://bazarr:6767"} 61021
# HELP bazarr_episode_subtitles_wanted_total Total number of wanted episode subtitles
# TYPE bazarr_episode_subtitles_wanted_total gauge
bazarr_episode_subtitles_wanted_total{url="http://bazarr:6767"} 18618
# HELP bazarr_movie_subtitles_downloaded_total Total number of downloaded movie subtitles
# TYPE bazarr_movie_subtitles_downloaded_total gauge
bazarr_movie_subtitles_downloaded_total{url="http://bazarr:6767"} 1973
# HELP bazarr_movie_subtitles_filesize_total Total filesize of all movie subtitles
# TYPE bazarr_movie_subtitles_filesize_total gauge
bazarr_movie_subtitles_filesize_total{url="http://bazarr:6767"} 2.42571861e+08
# HELP bazarr_movie_subtitles_history_total Total number of history movie subtitles
# TYPE bazarr_movie_subtitles_history_total gauge
bazarr_movie_subtitles_history_total{url="http://bazarr:6767"} 61
# HELP bazarr_movie_subtitles_missing_total Total number of missing movie subtitles
# TYPE bazarr_movie_subtitles_missing_total gauge
bazarr_movie_subtitles_missing_total{url="http://bazarr:6767"} 166
# HELP bazarr_movie_subtitles_monitored_total Total number of monitored movie subtitles
# TYPE bazarr_movie_subtitles_monitored_total gauge
bazarr_movie_subtitles_monitored_total{url="http://bazarr:6767"} 2126
# HELP bazarr_movie_subtitles_unmonitored_total Total number of unmonitored movie subtitles
# TYPE bazarr_movie_subtitles_unmonitored_total gauge
bazarr_movie_subtitles_unmonitored_total{url="http://bazarr:6767"} 5
# HELP bazarr_movie_subtitles_wanted_total Total number of wanted movie subtitles
# TYPE bazarr_movie_subtitles_wanted_total gauge
bazarr_movie_subtitles_wanted_total{url="http://bazarr:6767"} 166
# HELP bazarr_scrape_duration_seconds Duration of the last scrape of metrics from Exportarr.
# TYPE bazarr_scrape_duration_seconds gauge
bazarr_scrape_duration_seconds{url="http://bazarr:6767"} 28.936000262
# HELP bazarr_scrape_requests_total Total number of HTTP requests made.
# TYPE bazarr_scrape_requests_total counter
bazarr_scrape_requests_total{code="200",url="http://bazarr:6767"} 10
# HELP bazarr_subtitles_downloaded_total Total number of downloaded subtitles
# TYPE bazarr_subtitles_downloaded_total gauge
bazarr_subtitles_downloaded_total{url="http://bazarr:6767"} 66900
# HELP bazarr_subtitles_filesize_total Total filesize of all subtitles
# TYPE bazarr_subtitles_filesize_total gauge
bazarr_subtitles_filesize_total{url="http://bazarr:6767"} 3.498832673e+09
# HELP bazarr_subtitles_history_total Total number of history subtitles
# TYPE bazarr_subtitles_history_total gauge
bazarr_subtitles_history_total{url="http://bazarr:6767"} 18013
# HELP bazarr_subtitles_language_total Total number of downloaded subtitles by language
# TYPE bazarr_subtitles_language_total gauge
bazarr_subtitles_language_total{language="Arabic",url="http://bazarr:6767"} 51
bazarr_subtitles_language_total{language="Danish",url="http://bazarr:6767"} 41
bazarr_subtitles_language_total{language="Dutch",url="http://bazarr:6767"} 4
bazarr_subtitles_language_total{language="English",url="http://bazarr:6767"} 66628
bazarr_subtitles_language_total{language="Finnish",url="http://bazarr:6767"} 42
bazarr_subtitles_language_total{language="Hebrew",url="http://bazarr:6767"} 3
bazarr_subtitles_language_total{language="Italian",url="http://bazarr:6767"} 1
bazarr_subtitles_language_total{language="Norwegian",url="http://bazarr:6767"} 41
bazarr_subtitles_language_total{language="Polish",url="http://bazarr:6767"} 21
bazarr_subtitles_language_total{language="Portuguese",url="http://bazarr:6767"} 24
bazarr_subtitles_language_total{language="Romanian",url="http://bazarr:6767"} 1
bazarr_subtitles_language_total{language="Swedish",url="http://bazarr:6767"} 43
# HELP bazarr_subtitles_missing_total Total number of missing subtitles
# TYPE bazarr_subtitles_missing_total gauge
bazarr_subtitles_missing_total{url="http://bazarr:6767"} 18785
# HELP bazarr_subtitles_monitored_total Total number of monitored subtitles
# TYPE bazarr_subtitles_monitored_total gauge
bazarr_subtitles_monitored_total{url="http://bazarr:6767"} 29003
# HELP bazarr_subtitles_provider_total Total number of downloaded subtitles by provider
# TYPE bazarr_subtitles_provider_total gauge
bazarr_subtitles_provider_total{provider="betaseries",url="http://bazarr:6767"} 1486
bazarr_subtitles_provider_total{provider="bsplayer",url="http://bazarr:6767"} 52
bazarr_subtitles_provider_total{provider="manual",url="http://bazarr:6767"} 24
bazarr_subtitles_provider_total{provider="opensubtitles",url="http://bazarr:6767"} 6169
bazarr_subtitles_provider_total{provider="opensubtitlescom",url="http://bazarr:6767"} 763
bazarr_subtitles_provider_total{provider="podnapisi",url="http://bazarr:6767"} 3069
bazarr_subtitles_provider_total{provider="subscene",url="http://bazarr:6767"} 2
bazarr_subtitles_provider_total{provider="supersubtitles",url="http://bazarr:6767"} 1773
bazarr_subtitles_provider_total{provider="tusubtitulo",url="http://bazarr:6767"} 11
bazarr_subtitles_provider_total{provider="tvsubtitles",url="http://bazarr:6767"} 27
bazarr_subtitles_provider_total{provider="yifysubtitles",url="http://bazarr:6767"} 18
# HELP bazarr_subtitles_score_total Total number of downloaded subtitles by score
# TYPE bazarr_subtitles_score_total gauge
bazarr_subtitles_score_total{score="100.0%",url="http://bazarr:6767"} 1298
bazarr_subtitles_score_total{score="66.67%",url="http://bazarr:6767"} 9
bazarr_subtitles_score_total{score="66.94%",url="http://bazarr:6767"} 570
bazarr_subtitles_score_total{score="67.5%",url="http://bazarr:6767"} 58
bazarr_subtitles_score_total{score="67.78%",url="http://bazarr:6767"} 5
bazarr_subtitles_score_total{score="68.06%",url="http://bazarr:6767"} 1
bazarr_subtitles_score_total{score="68.33%",url="http://bazarr:6767"} 1
bazarr_subtitles_score_total{score="68.89%",url="http://bazarr:6767"} 29
bazarr_subtitles_score_total{score="69.44%",url="http://bazarr:6767"} 8
bazarr_subtitles_score_total{score="69.72%",url="http://bazarr:6767"} 5
bazarr_subtitles_score_total{score="70.28%",url="http://bazarr:6767"} 5
bazarr_subtitles_score_total{score="70.83%",url="http://bazarr:6767"} 1
bazarr_subtitles_score_total{score="71.11%",url="http://bazarr:6767"} 2
bazarr_subtitles_score_total{score="75.0%",url="http://bazarr:6767"} 4
bazarr_subtitles_score_total{score="75.83%",url="http://bazarr:6767"} 4
bazarr_subtitles_score_total{score="77.5%",url="http://bazarr:6767"} 4
bazarr_subtitles_score_total{score="78.33%",url="http://bazarr:6767"} 1
bazarr_subtitles_score_total{score="79.17%",url="http://bazarr:6767"} 5
bazarr_subtitles_score_total{score="80.0%",url="http://bazarr:6767"} 2
bazarr_subtitles_score_total{score="80.83%",url="http://bazarr:6767"} 1
bazarr_subtitles_score_total{score="81.67%",url="http://bazarr:6767"} 9
bazarr_subtitles_score_total{score="82.5%",url="http://bazarr:6767"} 2
bazarr_subtitles_score_total{score="83.33%",url="http://bazarr:6767"} 6
bazarr_subtitles_score_total{score="84.17%",url="http://bazarr:6767"} 1
bazarr_subtitles_score_total{score="85.0%",url="http://bazarr:6767"} 7
bazarr_subtitles_score_total{score="85.83%",url="http://bazarr:6767"} 7
bazarr_subtitles_score_total{score="87.5%",url="http://bazarr:6767"} 4
bazarr_subtitles_score_total{score="91.67%",url="http://bazarr:6767"} 115
bazarr_subtitles_score_total{score="91.94%",url="http://bazarr:6767"} 4723
bazarr_subtitles_score_total{score="92.22%",url="http://bazarr:6767"} 17
bazarr_subtitles_score_total{score="92.5%",url="http://bazarr:6767"} 991
bazarr_subtitles_score_total{score="92.78%",url="http://bazarr:6767"} 177
bazarr_subtitles_score_total{score="93.06%",url="http://bazarr:6767"} 84
bazarr_subtitles_score_total{score="93.33%",url="http://bazarr:6767"} 217
bazarr_subtitles_score_total{score="93.61%",url="http://bazarr:6767"} 27
bazarr_subtitles_score_total{score="93.89%",url="http://bazarr:6767"} 1382
bazarr_subtitles_score_total{score="94.17%",url="http://bazarr:6767"} 48
bazarr_subtitles_score_total{score="94.44%",url="http://bazarr:6767"} 1289
bazarr_subtitles_score_total{score="94.72%",url="http://bazarr:6767"} 77
bazarr_subtitles_score_total{score="95.0%",url="http://bazarr:6767"} 401
bazarr_subtitles_score_total{score="95.28%",url="http://bazarr:6767"} 132
bazarr_subtitles_score_total{score="95.83%",url="http://bazarr:6767"} 185
bazarr_subtitles_score_total{score="96.11%",url="http://bazarr:6767"} 24
bazarr_subtitles_score_total{score="96.67%",url="http://bazarr:6767"} 20
bazarr_subtitles_score_total{score="97.5%",url="http://bazarr:6767"} 2
bazarr_subtitles_score_total{score="98.06%",url="http://bazarr:6767"} 77
bazarr_subtitles_score_total{score="98.61%",url="http://bazarr:6767"} 889
bazarr_subtitles_score_total{score="98.89%",url="http://bazarr:6767"} 4
bazarr_subtitles_score_total{score="99.17%",url="http://bazarr:6767"} 360
bazarr_subtitles_score_total{score="99.44%",url="http://bazarr:6767"} 47
bazarr_subtitles_score_total{score="99.72%",url="http://bazarr:6767"} 57
# HELP bazarr_subtitles_unmonitored_total Total number of unmonitored subtitles
# TYPE bazarr_subtitles_unmonitored_total gauge
bazarr_subtitles_unmonitored_total{url="http://bazarr:6767"} 61026
# HELP bazarr_subtitles_wanted_total Total number of wanted subtitles
# TYPE bazarr_subtitles_wanted_total gauge
bazarr_subtitles_wanted_total{url="http://bazarr:6767"} 18784
# HELP bazarr_system_health_issues Total number of health issues by object and issue
# TYPE bazarr_system_health_issues gauge
bazarr_system_health_issues{issue="",object="",url="http://bazarr:6767"} 0
# HELP bazarr_system_status System Status
# TYPE bazarr_system_status gauge
bazarr_system_status{url="http://bazarr:6767"} 1
# HELP exportarr_app_info A metric with a constant '1' value labeled by app name, version, build time, and revision.
# TYPE exportarr_app_info gauge
exportarr_app_info{app_name="exportarr",build_time="2023-10-18T12:55:57.008Z",revision="294191ecae08120e407f1a8b2c5dda12ffb7b007",version="v1.6.0"} 1

new:

# HELP bazarr_episode_subtitles_downloaded_total Total number of downloaded episode subtitles
# TYPE bazarr_episode_subtitles_downloaded_total gauge
bazarr_episode_subtitles_downloaded_total{url="http://bazarr:6767"} 64927
# HELP bazarr_episode_subtitles_filesize_total Total filesize of all episode subtitles
# TYPE bazarr_episode_subtitles_filesize_total gauge
bazarr_episode_subtitles_filesize_total{url="http://bazarr:6767"} 3.256260812e+09
# HELP bazarr_episode_subtitles_history_total Total number of history episode subtitles
# TYPE bazarr_episode_subtitles_history_total gauge
bazarr_episode_subtitles_history_total{url="http://bazarr:6767"} 17952
# HELP bazarr_episode_subtitles_missing_total Total number of missing episode subtitles
# TYPE bazarr_episode_subtitles_missing_total gauge
bazarr_episode_subtitles_missing_total{url="http://bazarr:6767"} 18619
# HELP bazarr_episode_subtitles_monitored_total Total number of monitored episode subtitles
# TYPE bazarr_episode_subtitles_monitored_total gauge
bazarr_episode_subtitles_monitored_total{url="http://bazarr:6767"} 26877
# HELP bazarr_episode_subtitles_unmonitored_total Total number of unmonitored episode subtitles
# TYPE bazarr_episode_subtitles_unmonitored_total gauge
bazarr_episode_subtitles_unmonitored_total{url="http://bazarr:6767"} 61021
# HELP bazarr_episode_subtitles_wanted_total Total number of wanted episode subtitles
# TYPE bazarr_episode_subtitles_wanted_total gauge
bazarr_episode_subtitles_wanted_total{url="http://bazarr:6767"} 18618
# HELP bazarr_movie_subtitles_downloaded_total Total number of downloaded movie subtitles
# TYPE bazarr_movie_subtitles_downloaded_total gauge
bazarr_movie_subtitles_downloaded_total{url="http://bazarr:6767"} 1973
# HELP bazarr_movie_subtitles_filesize_total Total filesize of all movie subtitles
# TYPE bazarr_movie_subtitles_filesize_total gauge
bazarr_movie_subtitles_filesize_total{url="http://bazarr:6767"} 2.42571861e+08
# HELP bazarr_movie_subtitles_history_total Total number of history movie subtitles
# TYPE bazarr_movie_subtitles_history_total gauge
bazarr_movie_subtitles_history_total{url="http://bazarr:6767"} 61
# HELP bazarr_movie_subtitles_missing_total Total number of missing movie subtitles
# TYPE bazarr_movie_subtitles_missing_total gauge
bazarr_movie_subtitles_missing_total{url="http://bazarr:6767"} 166
# HELP bazarr_movie_subtitles_monitored_total Total number of monitored movie subtitles
# TYPE bazarr_movie_subtitles_monitored_total gauge
bazarr_movie_subtitles_monitored_total{url="http://bazarr:6767"} 2126
# HELP bazarr_movie_subtitles_unmonitored_total Total number of unmonitored movie subtitles
# TYPE bazarr_movie_subtitles_unmonitored_total gauge
bazarr_movie_subtitles_unmonitored_total{url="http://bazarr:6767"} 5
# HELP bazarr_movie_subtitles_wanted_total Total number of wanted movie subtitles
# TYPE bazarr_movie_subtitles_wanted_total gauge
bazarr_movie_subtitles_wanted_total{url="http://bazarr:6767"} 166
# HELP bazarr_scrape_duration_seconds Duration of the last scrape of metrics from Exportarr.
# TYPE bazarr_scrape_duration_seconds gauge
bazarr_scrape_duration_seconds{url="http://bazarr:6767"} 25.886018311
# HELP bazarr_scrape_requests_total Total number of HTTP requests made.
# TYPE bazarr_scrape_requests_total counter
bazarr_scrape_requests_total{code="200",url="http://bazarr:6767"} 6
# HELP bazarr_subtitles_downloaded_total Total number of downloaded subtitles
# TYPE bazarr_subtitles_downloaded_total gauge
bazarr_subtitles_downloaded_total{url="http://bazarr:6767"} 66900
# HELP bazarr_subtitles_filesize_total Total filesize of all subtitles
# TYPE bazarr_subtitles_filesize_total gauge
bazarr_subtitles_filesize_total{url="http://bazarr:6767"} 3.498832673e+09
# HELP bazarr_subtitles_history_total Total number of history subtitles
# TYPE bazarr_subtitles_history_total gauge
bazarr_subtitles_history_total{url="http://bazarr:6767"} 18013
# HELP bazarr_subtitles_language_total Total number of downloaded subtitles by language
# TYPE bazarr_subtitles_language_total gauge
bazarr_subtitles_language_total{language="Arabic",url="http://bazarr:6767"} 51
bazarr_subtitles_language_total{language="Danish",url="http://bazarr:6767"} 41
bazarr_subtitles_language_total{language="Dutch",url="http://bazarr:6767"} 4
bazarr_subtitles_language_total{language="English",url="http://bazarr:6767"} 66628
bazarr_subtitles_language_total{language="Finnish",url="http://bazarr:6767"} 42
bazarr_subtitles_language_total{language="Hebrew",url="http://bazarr:6767"} 3
bazarr_subtitles_language_total{language="Italian",url="http://bazarr:6767"} 1
bazarr_subtitles_language_total{language="Norwegian",url="http://bazarr:6767"} 41
bazarr_subtitles_language_total{language="Polish",url="http://bazarr:6767"} 21
bazarr_subtitles_language_total{language="Portuguese",url="http://bazarr:6767"} 24
bazarr_subtitles_language_total{language="Romanian",url="http://bazarr:6767"} 1
bazarr_subtitles_language_total{language="Swedish",url="http://bazarr:6767"} 43
# HELP bazarr_subtitles_missing_total Total number of missing subtitles
# TYPE bazarr_subtitles_missing_total gauge
bazarr_subtitles_missing_total{url="http://bazarr:6767"} 18785
# HELP bazarr_subtitles_monitored_total Total number of monitored subtitles
# TYPE bazarr_subtitles_monitored_total gauge
bazarr_subtitles_monitored_total{url="http://bazarr:6767"} 29003
# HELP bazarr_subtitles_provider_total Total number of downloaded subtitles by provider
# TYPE bazarr_subtitles_provider_total gauge
bazarr_subtitles_provider_total{provider="betaseries",url="http://bazarr:6767"} 1486
bazarr_subtitles_provider_total{provider="bsplayer",url="http://bazarr:6767"} 52
bazarr_subtitles_provider_total{provider="manual",url="http://bazarr:6767"} 24
bazarr_subtitles_provider_total{provider="opensubtitles",url="http://bazarr:6767"} 6169
bazarr_subtitles_provider_total{provider="opensubtitlescom",url="http://bazarr:6767"} 763
bazarr_subtitles_provider_total{provider="podnapisi",url="http://bazarr:6767"} 3069
bazarr_subtitles_provider_total{provider="subscene",url="http://bazarr:6767"} 2
bazarr_subtitles_provider_total{provider="supersubtitles",url="http://bazarr:6767"} 1773
bazarr_subtitles_provider_total{provider="tusubtitulo",url="http://bazarr:6767"} 11
bazarr_subtitles_provider_total{provider="tvsubtitles",url="http://bazarr:6767"} 27
bazarr_subtitles_provider_total{provider="yifysubtitles",url="http://bazarr:6767"} 18
# HELP bazarr_subtitles_score_total Total number of downloaded subtitles by score
# TYPE bazarr_subtitles_score_total gauge
bazarr_subtitles_score_total{score="100.0%",url="http://bazarr:6767"} 1298
bazarr_subtitles_score_total{score="66.67%",url="http://bazarr:6767"} 9
bazarr_subtitles_score_total{score="66.94%",url="http://bazarr:6767"} 570
bazarr_subtitles_score_total{score="67.5%",url="http://bazarr:6767"} 58
bazarr_subtitles_score_total{score="67.78%",url="http://bazarr:6767"} 5
bazarr_subtitles_score_total{score="68.06%",url="http://bazarr:6767"} 1
bazarr_subtitles_score_total{score="68.33%",url="http://bazarr:6767"} 1
bazarr_subtitles_score_total{score="68.89%",url="http://bazarr:6767"} 29
bazarr_subtitles_score_total{score="69.44%",url="http://bazarr:6767"} 8
bazarr_subtitles_score_total{score="69.72%",url="http://bazarr:6767"} 5
bazarr_subtitles_score_total{score="70.28%",url="http://bazarr:6767"} 5
bazarr_subtitles_score_total{score="70.83%",url="http://bazarr:6767"} 1
bazarr_subtitles_score_total{score="71.11%",url="http://bazarr:6767"} 2
bazarr_subtitles_score_total{score="75.0%",url="http://bazarr:6767"} 4
bazarr_subtitles_score_total{score="75.83%",url="http://bazarr:6767"} 4
bazarr_subtitles_score_total{score="77.5%",url="http://bazarr:6767"} 4
bazarr_subtitles_score_total{score="78.33%",url="http://bazarr:6767"} 1
bazarr_subtitles_score_total{score="79.17%",url="http://bazarr:6767"} 5
bazarr_subtitles_score_total{score="80.0%",url="http://bazarr:6767"} 2
bazarr_subtitles_score_total{score="80.83%",url="http://bazarr:6767"} 1
bazarr_subtitles_score_total{score="81.67%",url="http://bazarr:6767"} 9
bazarr_subtitles_score_total{score="82.5%",url="http://bazarr:6767"} 2
bazarr_subtitles_score_total{score="83.33%",url="http://bazarr:6767"} 6
bazarr_subtitles_score_total{score="84.17%",url="http://bazarr:6767"} 1
bazarr_subtitles_score_total{score="85.0%",url="http://bazarr:6767"} 7
bazarr_subtitles_score_total{score="85.83%",url="http://bazarr:6767"} 7
bazarr_subtitles_score_total{score="87.5%",url="http://bazarr:6767"} 4
bazarr_subtitles_score_total{score="91.67%",url="http://bazarr:6767"} 115
bazarr_subtitles_score_total{score="91.94%",url="http://bazarr:6767"} 4723
bazarr_subtitles_score_total{score="92.22%",url="http://bazarr:6767"} 17
bazarr_subtitles_score_total{score="92.5%",url="http://bazarr:6767"} 991
bazarr_subtitles_score_total{score="92.78%",url="http://bazarr:6767"} 177
bazarr_subtitles_score_total{score="93.06%",url="http://bazarr:6767"} 84
bazarr_subtitles_score_total{score="93.33%",url="http://bazarr:6767"} 217
bazarr_subtitles_score_total{score="93.61%",url="http://bazarr:6767"} 27
bazarr_subtitles_score_total{score="93.89%",url="http://bazarr:6767"} 1382
bazarr_subtitles_score_total{score="94.17%",url="http://bazarr:6767"} 48
bazarr_subtitles_score_total{score="94.44%",url="http://bazarr:6767"} 1289
bazarr_subtitles_score_total{score="94.72%",url="http://bazarr:6767"} 77
bazarr_subtitles_score_total{score="95.0%",url="http://bazarr:6767"} 401
bazarr_subtitles_score_total{score="95.28%",url="http://bazarr:6767"} 132
bazarr_subtitles_score_total{score="95.83%",url="http://bazarr:6767"} 185
bazarr_subtitles_score_total{score="96.11%",url="http://bazarr:6767"} 24
bazarr_subtitles_score_total{score="96.67%",url="http://bazarr:6767"} 20
bazarr_subtitles_score_total{score="97.5%",url="http://bazarr:6767"} 2
bazarr_subtitles_score_total{score="98.06%",url="http://bazarr:6767"} 77
bazarr_subtitles_score_total{score="98.61%",url="http://bazarr:6767"} 889
bazarr_subtitles_score_total{score="98.89%",url="http://bazarr:6767"} 4
bazarr_subtitles_score_total{score="99.17%",url="http://bazarr:6767"} 360
bazarr_subtitles_score_total{score="99.44%",url="http://bazarr:6767"} 47
bazarr_subtitles_score_total{score="99.72%",url="http://bazarr:6767"} 57
# HELP bazarr_subtitles_unmonitored_total Total number of unmonitored subtitles
# TYPE bazarr_subtitles_unmonitored_total gauge
bazarr_subtitles_unmonitored_total{url="http://bazarr:6767"} 61026
# HELP bazarr_subtitles_wanted_total Total number of wanted subtitles
# TYPE bazarr_subtitles_wanted_total gauge
bazarr_subtitles_wanted_total{url="http://bazarr:6767"} 18784
# HELP bazarr_system_health_issues Total number of health issues by object and issue
# TYPE bazarr_system_health_issues gauge
bazarr_system_health_issues{issue="",object="",url="http://bazarr:6767"} 0
# HELP bazarr_system_status System Status
# TYPE bazarr_system_status gauge
bazarr_system_status{url="http://bazarr:6767"} 1
# HELP exportarr_app_info A metric with a constant '1' value labeled by app name, version, build time, and revision.
# TYPE exportarr_app_info gauge
exportarr_app_info{app_name="exportarr",build_time="",revision="",version="development"} 1

phyzical · 2023-10-30T08:08:29Z

so yeah its actually slower, which is something i noticed when i tried batching it originally, granted i didn't try hard to workout why

i did my best to be fair too. 1 req at a time randomly switching between who was next and no prometheus running causing additional random load.

Logic wise its the same though

rtrox · 2023-11-02T04:55:55Z

The caching is it a simple toggleable feature or is it fairly deep? it does feel like an edgecase at this point i would personally consider my library the large end but this other users has double what i do so 😆 (no saying it shouldnt be a feature but should depend on how complicated it becomes to support optional caching)

It's tied into the prowlarr logic, but it's also not terribly complicated. You can see the indexerStatCache here. That said, I only did caching there due to the shape of the data returned by prowlarr (their API accepts a time range, and gives aggregated stats by day. So we have to cache last seen values to calculate differentials).

so yeah its actually slower, which is something i noticed when i tried batching it originally, granted i didn't try hard to workout why

It's probably just the sheer number of calls. I set the default concurrency limit to 10 inflight requests. You can test bumping that up to 20 or 30 by setting --series-batch-concurrency. You can also bump up the batch size with --series-batch-size (default 50). If bumping either of those up makes a difference I'll just change the defaults. I should also add a collector stat for the number of calls being sent as well.

If it's still not faster, I think I'll still merge it, it looks like it's about a ~10% regression, which is likely worth it to reduce the size of the API requests. But if we can't actually get faster, I'll likely put the iterative call behind EnableAdditionalMetrics so folks can decide whether they want to see less metrics, or have a higher poll rate.

(@onedr0p feel free to veto me on that)

onedr0p · 2023-11-02T14:11:33Z

But if we can't actually get faster, I'll likely put the iterative call behind EnableAdditionalMetrics so folks can decide whether they want to see less metrics, or have a higher poll rate.

Sounds good to me :)

phyzical · 2023-11-03T01:18:09Z

@rtrox ill give a few more tests around --series-batch-concurrency tonight after work 👍

But i think leveraging "EnableAdditionalMetrics" for sync v async i think is fair, though may be a bit misleading

But i think if it does actually allow users to use the implementation i think its a good idea.

rtrox · 2023-11-03T02:22:07Z

@phyzical

But i think leveraging "EnableAdditionalMetrics" for sync v async i think is fair, though may be a bit misleading

That's not quite what I mean -- either way I'll want to switch to the threaded requests here, I'm not super comfortable leaving the episode history calls in place as is, the URLs are too large for me to be comfortable with on the wire, and the memory required for the responses is concerning. So I'm actually ok with a wall clock regression to move to the threaded model where request size is bounded.

What I mean is -- if we can't find defaults that let us reduce the wall clock time, I plan to put that call and the metrics it generates behind EnableAdditionalMetrics, I've been looking through the API code in bazarr, it looks like we can get a pretty reasonable number of metrics directly from the /series endpoint (episode files downloaded, episode files missing, series monitored.

So let's see if we can find some defaults that boost wall clock time, otherwise I'll start some refactoring to pull more metrics from the series api, and put the episode stats calls behind a flag (I'll likely do the refactoring either way, as any metric we can generate from the pre-aggregated series data is going to be much more performant. There are some other changes, like #239 that I'll want to make anyway).

phyzical · 2023-11-03T03:59:32Z

Ah yep, sorry i get you now. i think your safe to just chuck both the history blocks behind the flag, i believe i didn't just cause i was focusing on timings and not ram usage.

I don't think you can get the data found in history in the main api call (which is why i started using it) but i could be wrong.

phyzical · 2023-11-09T10:17:42Z

ah sorry, life then work got in the way 😆

batch concurrency = 10 is what was being used above

~40mb
~24s

batch concurrency = 20

~42mb
~24-25s

batch concurrency = 30

~45-50mb
~24-25s

batch concurrency = 40

~50mb
~33s

batch concurrency = 50

~50mb
~33-34

its odd that things get slower the more concurrency i wonder if its more a limitation bazarr side? as in how well the python code can deal with concurrency vs a giant payload.

ill do a few more tests playing with the batch size now for some more numbers

phyzical · 2023-11-09T10:38:24Z

batch-size 50 is the first one above

batch-size 100

~50 mb
~28s

batch-size 150

~50mb
~27s

batch-size 200

~28s
~50mb

batch-size 300

~21s
~70-100mb

batch-size 500

~21s
~100mb

phyzical · 2023-11-09T10:41:20Z

So almost all the attempts floated around the 50mb range until i increased the batch size which makes sense.

But i think the "recommendation" should be increase batch to reduce ram usage at the cost of longer requests, increase batch size for faster responses at the cost of more ram usage.

personally i think ill just configure it to, have a batch size of 300-500 and use the default concurrency.

@rtrox feel free to list some combos of vars you want the metrics of if it helps at all, and i will do some more tests

rtrox · 2023-11-09T19:09:27Z

Thanks @phyzical, great stuff! Yea, I'd suspect the same. I'd guess either Bazarr, SQLite, or query performance as the issue, but this is probably good enough, 20-30 seconds at your library size seems reasonably solid.

So to land this, I think two updates:

I'll update the defaults to 300 batch size.
I'll put the history calls behind EnableAdditionalMetrics
[stretch] I may add some series level metrics, to ensure we can get some missing/monitored data for sonarr even without EnableAdditionalMetrics.

I should be be able to get to that over the long weekend so we can land this early next week.

phyzical · 2023-11-10T00:08:54Z

sounds good to me 👍

Signed-off-by: Russell <russell@troxel.io>

…tionalMetrics Signed-off-by: Russell Troxel <russell@troxel.io>

rtrox · 2023-11-29T20:56:53Z

Sorry, took a bit to get back to this PR, life got hectic >.< I think this should be ready for final review+merge.

phyzical · 2023-11-30T02:34:49Z

no worries, ill give it a scan and test agaisnt my instance when i can

phyzical · 2024-01-16T09:39:23Z

ah sorry i forgot then the holidays happened, i will give this a look this week

phyzical · 2024-01-25T07:37:00Z

was able to do a proper benchmark comparison diff wise for information it is exactly the same with the EnableAdditionalMetrics flag

👍

phyzical · 2024-01-25T07:39:07Z

@onedr0p when you have time, i think this is good to be merged

….1@60cf3d4 by renovate (#17725) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [ghcr.io/onedr0p/exportarr](https://togithub.com/onedr0p/exportarr) | patch | `v1.6.0` -> `v1.6.1` | --- > [!WARNING] > Some dependencies could not be looked up. Check the Dependency Dashboard for more information. --- ### Release Notes <details> <summary>onedr0p/exportarr (ghcr.io/onedr0p/exportarr)</summary> ### [`v1.6.1`](https://togithub.com/onedr0p/exportarr/releases/tag/v1.6.1) [Compare Source](https://togithub.com/onedr0p/exportarr/compare/v1.6.0...v1.6.1) ##### What's Changed - Add tests for Radarr by [@rtrox](https://togithub.com/rtrox) in [onedr0p/exportarr#234 - Add tests for Sonarr by [@rtrox](https://togithub.com/rtrox) in [onedr0p/exportarr#235 - Add tests for Readarr by [@rtrox](https://togithub.com/rtrox) in [onedr0p/exportarr#236 - doc(README): remove typo by [@Deep145757](https://togithub.com/Deep145757) in [onedr0p/exportarr#240 - chore(deps): update golang docker tag to v1.21.4 by [@renovate](https://togithub.com/renovate) in [onedr0p/exportarr#242 - chore(deps): update actions/setup-go action to v5 by [@renovate](https://togithub.com/renovate) in [onedr0p/exportarr#248 - chore(deps): update golang docker tag to v1.21.5 by [@renovate](https://togithub.com/renovate) in [onedr0p/exportarr#247 - chore(deps): update actions/checkout digest to [`b4ffde6`](https://togithub.com/onedr0p/exportarr/commit/b4ffde6) by [@renovate](https://togithub.com/renovate) in [onedr0p/exportarr#233 - \[bazarr] Add bounded concurrency to episodes endpoint by [@rtrox](https://togithub.com/rtrox) in [onedr0p/exportarr#238 - chore(deps): update golangci/golangci-lint-action action to v3 by [@renovate](https://togithub.com/renovate) in [onedr0p/exportarr#257 - fix: add `gomodTidy` option to renovate by [@rtrox](https://togithub.com/rtrox) in [onedr0p/exportarr#258 - fix(deps): update module github.com/gookit/validate to v1.5.2 by [@renovate](https://togithub.com/renovate) in [onedr0p/exportarr#253 - chore(deps): update actions/setup-go action to v5 by [@renovate](https://togithub.com/renovate) in [onedr0p/exportarr#256 - chore(deps): update actions/checkout digest to [`b4ffde6`](https://togithub.com/onedr0p/exportarr/commit/b4ffde6) by [@renovate](https://togithub.com/renovate) in [onedr0p/exportarr#255 - fix(deps): update module github.com/prometheus/client_golang to v1.18.0 by [@renovate](https://togithub.com/renovate) in [onedr0p/exportarr#250 - fix(deps): update module golang.org/x/sync to v0.6.0 by [@renovate](https://togithub.com/renovate) in [onedr0p/exportarr#220 - fix(deps): update module github.com/spf13/cobra to v1.8.0 by [@renovate](https://togithub.com/renovate) in [onedr0p/exportarr#241 - fix(deps): update golang.org/x/exp digest to [`1b97071`](https://togithub.com/onedr0p/exportarr/commit/1b97071) by [@renovate](https://togithub.com/renovate) in [onedr0p/exportarr#243 - chore(deps): update golang docker tag to v1.21.6 by [@renovate](https://togithub.com/renovate) in [onedr0p/exportarr#251 - fix([#252](https://togithub.com/onedr0p/exportarr/issues/252)): Handle an empty Server Stat map returned from Sab. by [@rtrox](https://togithub.com/rtrox) in [onedr0p/exportarr#259 ##### New Contributors - [@Deep145757](https://togithub.com/Deep145757) made their first contribution in [onedr0p/exportarr#240 **Full Changelog**: onedr0p/exportarr@v1.6.0...v1.6.1 </details> --- ### Configuration 📅 **Schedule**: Branch creation - "before 10pm on monday" in timezone Europe/Amsterdam, Automerge - At any time (no schedule defined). 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate).

….1@60cf3d4 by renovate (truecharts#17725) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [ghcr.io/onedr0p/exportarr](https://togithub.com/onedr0p/exportarr) | patch | `v1.6.0` -> `v1.6.1` | --- > [!WARNING] > Some dependencies could not be looked up. Check the Dependency Dashboard for more information. --- ### Release Notes <details> <summary>onedr0p/exportarr (ghcr.io/onedr0p/exportarr)</summary> ### [`v1.6.1`](https://togithub.com/onedr0p/exportarr/releases/tag/v1.6.1) [Compare Source](https://togithub.com/onedr0p/exportarr/compare/v1.6.0...v1.6.1) ##### What's Changed - Add tests for Radarr by [@&truecharts#8203;rtrox](https://togithub.com/rtrox) in [onedr0p/exportarr#234 - Add tests for Sonarr by [@&truecharts#8203;rtrox](https://togithub.com/rtrox) in [onedr0p/exportarr#235 - Add tests for Readarr by [@&truecharts#8203;rtrox](https://togithub.com/rtrox) in [onedr0p/exportarr#236 - doc(README): remove typo by [@&truecharts#8203;Deep145757](https://togithub.com/Deep145757) in [onedr0p/exportarr#240 - chore(deps): update golang docker tag to v1.21.4 by [@&truecharts#8203;renovate](https://togithub.com/renovate) in [onedr0p/exportarr#242 - chore(deps): update actions/setup-go action to v5 by [@&truecharts#8203;renovate](https://togithub.com/renovate) in [onedr0p/exportarr#248 - chore(deps): update golang docker tag to v1.21.5 by [@&truecharts#8203;renovate](https://togithub.com/renovate) in [onedr0p/exportarr#247 - chore(deps): update actions/checkout digest to [`b4ffde6`](https://togithub.com/onedr0p/exportarr/commit/b4ffde6) by [@&truecharts#8203;renovate](https://togithub.com/renovate) in [onedr0p/exportarr#233 - \[bazarr] Add bounded concurrency to episodes endpoint by [@&truecharts#8203;rtrox](https://togithub.com/rtrox) in [onedr0p/exportarr#238 - chore(deps): update golangci/golangci-lint-action action to v3 by [@&truecharts#8203;renovate](https://togithub.com/renovate) in [onedr0p/exportarr#257 - fix: add `gomodTidy` option to renovate by [@&truecharts#8203;rtrox](https://togithub.com/rtrox) in [onedr0p/exportarr#258 - fix(deps): update module github.com/gookit/validate to v1.5.2 by [@&truecharts#8203;renovate](https://togithub.com/renovate) in [onedr0p/exportarr#253 - chore(deps): update actions/setup-go action to v5 by [@&truecharts#8203;renovate](https://togithub.com/renovate) in [onedr0p/exportarr#256 - chore(deps): update actions/checkout digest to [`b4ffde6`](https://togithub.com/onedr0p/exportarr/commit/b4ffde6) by [@&truecharts#8203;renovate](https://togithub.com/renovate) in [onedr0p/exportarr#255 - fix(deps): update module github.com/prometheus/client_golang to v1.18.0 by [@&truecharts#8203;renovate](https://togithub.com/renovate) in [onedr0p/exportarr#250 - fix(deps): update module golang.org/x/sync to v0.6.0 by [@&truecharts#8203;renovate](https://togithub.com/renovate) in [onedr0p/exportarr#220 - fix(deps): update module github.com/spf13/cobra to v1.8.0 by [@&truecharts#8203;renovate](https://togithub.com/renovate) in [onedr0p/exportarr#241 - fix(deps): update golang.org/x/exp digest to [`1b97071`](https://togithub.com/onedr0p/exportarr/commit/1b97071) by [@&truecharts#8203;renovate](https://togithub.com/renovate) in [onedr0p/exportarr#243 - chore(deps): update golang docker tag to v1.21.6 by [@&truecharts#8203;renovate](https://togithub.com/renovate) in [onedr0p/exportarr#251 - fix([#&truecharts#8203;252](https://togithub.com/onedr0p/exportarr/issues/252)): Handle an empty Server Stat map returned from Sab. by [@&truecharts#8203;rtrox](https://togithub.com/rtrox) in [onedr0p/exportarr#259 ##### New Contributors - [@&truecharts#8203;Deep145757](https://togithub.com/Deep145757) made their first contribution in [onedr0p/exportarr#240 **Full Changelog**: onedr0p/exportarr@v1.6.0...v1.6.1 </details> --- ### Configuration 📅 **Schedule**: Branch creation - "before 10pm on monday" in timezone Europe/Amsterdam, Automerge - At any time (no schedule defined). 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate).

rtrox requested a review from onedr0p as a code owner October 29, 2023 00:49

rtrox added 5 commits November 29, 2023 12:51

[bazarr] Add bounded concurrency to episodes endpoint

b662ceb

Signed-off-by: Russell <russell@troxel.io>

Plumb Bazarr Config & Add defaults

047d7b0

Signed-off-by: Russell <russell@troxel.io>

Move GolangCILint to run first

97855e8

Signed-off-by: Russell <russell@troxel.io>

break linter out into it's own job

2e89131

Signed-off-by: Russell <russell@troxel.io>

Bump default batch size to 300, put episode metrics behind EnableAddi…

84760de

…tionalMetrics Signed-off-by: Russell Troxel <russell@troxel.io>

rtrox force-pushed the bazarr_concurrency branch from 4df37c6 to 84760de Compare November 29, 2023 20:52

phyzical approved these changes Jan 25, 2024

View reviewed changes

phyzical mentioned this pull request Jan 25, 2024

Bazarr: error when there are no series #244

Open

onedr0p approved these changes Jan 25, 2024

View reviewed changes

onedr0p merged commit 3276edb into master Jan 25, 2024
2 checks passed

onedr0p deleted the bazarr_concurrency branch March 29, 2024 12:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[bazarr] Add bounded concurrency to episodes endpoint #238

[bazarr] Add bounded concurrency to episodes endpoint #238

rtrox commented Oct 29, 2023

phyzical commented Oct 30, 2023 •

edited

rtrox commented Oct 30, 2023

rtrox commented Oct 30, 2023

phyzical commented Oct 30, 2023

rtrox commented Oct 30, 2023

phyzical commented Oct 30, 2023

rtrox commented Oct 30, 2023 •

edited

phyzical commented Oct 30, 2023

phyzical commented Oct 30, 2023 •

edited

rtrox commented Nov 2, 2023

onedr0p commented Nov 2, 2023

phyzical commented Nov 3, 2023

rtrox commented Nov 3, 2023 •

edited

phyzical commented Nov 3, 2023 •

edited

phyzical commented Nov 9, 2023

phyzical commented Nov 9, 2023

phyzical commented Nov 9, 2023 •

edited

rtrox commented Nov 9, 2023

phyzical commented Nov 10, 2023

rtrox commented Nov 29, 2023

phyzical commented Nov 30, 2023

phyzical commented Jan 16, 2024

phyzical commented Jan 25, 2024

phyzical commented Jan 25, 2024

[bazarr] Add bounded concurrency to episodes endpoint #238

[bazarr] Add bounded concurrency to episodes endpoint #238

Conversation

rtrox commented Oct 29, 2023

phyzical commented Oct 30, 2023 • edited

rtrox commented Oct 30, 2023

rtrox commented Oct 30, 2023

phyzical commented Oct 30, 2023

rtrox commented Oct 30, 2023

phyzical commented Oct 30, 2023

rtrox commented Oct 30, 2023 • edited

phyzical commented Oct 30, 2023

phyzical commented Oct 30, 2023 • edited

rtrox commented Nov 2, 2023

onedr0p commented Nov 2, 2023

phyzical commented Nov 3, 2023

rtrox commented Nov 3, 2023 • edited

phyzical commented Nov 3, 2023 • edited

phyzical commented Nov 9, 2023

phyzical commented Nov 9, 2023

phyzical commented Nov 9, 2023 • edited

rtrox commented Nov 9, 2023

phyzical commented Nov 10, 2023

rtrox commented Nov 29, 2023

phyzical commented Nov 30, 2023

phyzical commented Jan 16, 2024

phyzical commented Jan 25, 2024

phyzical commented Jan 25, 2024

phyzical commented Oct 30, 2023 •

edited

rtrox commented Oct 30, 2023 •

edited

phyzical commented Oct 30, 2023 •

edited

rtrox commented Nov 3, 2023 •

edited

phyzical commented Nov 3, 2023 •

edited

phyzical commented Nov 9, 2023 •

edited