Influxdb2 (#3)

* Boerderij#203 * Update docker compose to specify influxdb:1.8.4 * Update requirements to use urllib3==1.26.5 * updated to support Radarr and Sonarr V3 Api * bump requirements for requests * Fix Sonarr & Radarr V3 API /queue endpoint (Boerderij#220) * Fix lint issues * More lint fixes * Update Sonarr structures * Add Overseerr Support (Boerderij#210) * Remove duplicate structures * update changelog to reflect v1.7.7 changes * Add IP data to tautulli Boerderij#202 * add missing ip address in tautulli * Fixed: Streamlined API calls to Radarr and Sonarr (Boerderij#221) * Fixed: Sonarr Data pull issues (Boerderij#222) * Fix Sonarrr calendar * Update lidarr structure (Boerderij#225) Added missing arguments to Lidarr structure Fixes Boerderij#223 * Clean up request totals. Upstream change sct/overseerr#2426 * Cleanup blank space * Fix requested_date syntax. * Fix requested_date for Overseerr tv and movie * Fix overseerr config refernces * Fix overseerr structures * Update intparser to accommodate changes to config structure * Cleanup overseerr data collection * Fix SERVICES_ENABLED in varken.py to acomidate overseerr * Fixed: Sonarr/Lidarr Queues (Boerderij#227) * Change sonarr queue structures to str * Fixed: Multipage queue fetching * Update historical tautulli import (Boerderij#226) * Fixed: Sonarr perams ordering * Fixed: Proper warnings for missing data in sonarr and radarr * Added: Overseerr ENVs to docker compose. * Added: Logging to empty/no data returns * Update Sonarr & Lidarr Structs to match latest API changes (Boerderij#231) * Add support for estimatedCompletionTime in LidarrQueue * Add support for tvdbId in SonarrEpisode struct * Fix typo in docker yml * Rename example url for overseerr in docker yml * Update radarr structures to inclue originalLanguage * Update radarr structures to include addOptions * Update radarr structures to include popularity * fix(ombi): Update structures.py (Boerderij#238) * feat(docker): remove envs from example * fix(logging): remove depreciation warning. Var for debug mode (Boerderij#240) * fix(build): bump schedule version to 1.1 * fix(build): bump docker python version * fix(dep): update requests and urllib3 * fix(sonarr): ensure invalid sonarr queue items are just skipped over - fixes Boerderij#239 (Boerderij#243) * add branch to build inputs * update pipeline badge * Update automation * Add influxdb 2 client * Add structure for influxdb 2 params This contains all the data needed for connecting and writing to an InfluxDB2 server * Parse influxdb 2 config data * Add influxdb2 manager class This stores the data needed for InfluxDB2, and has a single `write_points` function on this that takes an array of points to add to the database * Use the correct db manager for varken * Add influxdb2 to the example varken config file * Create influx bucket if it doesn't exist * Update InfluxDB type on README * Clean up linting errors * Wrap create bucket in try/catch * Use bucket given in ini file * Log exception to troubleshoot errors * Allow configured influx2 address as URL (no port) * Bypass validity check to troubleshoot --------- Co-authored-by: mal5305 <malcolm.e.rogers@gmail.com> Co-authored-by: samwiseg0 <2241731+samwiseg0@users.noreply.github.com> Co-authored-by: Robin <19610103+RobinDadswell@users.noreply.github.com> Co-authored-by: tigattack <10629864+tigattack@users.noreply.github.com> Co-authored-by: Stewart Thomson <stewartthomson3@gmail.com> Co-authored-by: Cameron Stephen <mail@cajs.co.uk> Co-authored-by: MDHMatt <10845262+MDHMatt@users.noreply.github.com> Co-authored-by: Nathan Adams <dinnerbone@dinnerbone.com> Co-authored-by: Nicholas St. Germain <nick@cajun.pro> Co-authored-by: Gabe Revells <gcrevell@mtu.edu>
d-mcknight · Jun 23, 2023 · 7e03144 · 7e03144
1 parent b5a83f0
commit 7e03144
Show file tree

Hide file tree

Showing 15 changed files with 578 additions and 194 deletions.
diff --git a/Dockerfile b/Dockerfile
@@ -1,4 +1,4 @@
-FROM python:3.9.1-alpine
+FROM python:3.10.5-alpine
 
 ENV DEBUG="True" \
     DATA_FOLDER="/config" \

diff --git a/README.md b/README.md
@@ -17,7 +17,7 @@ ecosystem into InfluxDB using Grafana for a frontend
 Requirements:
 * [Python 3.6.7+](https://www.python.org/downloads/release/python-367/)
 * [Python3-pip](https://pip.pypa.io/en/stable/installing/)
-* [InfluxDB 1.8.x](https://www.influxdata.com/)
+* [InfluxDB 1.8.x or 2.0.x](https://www.influxdata.com/)
 * [Grafana](https://grafana.com/)
 
 <p align="center">

diff --git a/Varken.py b/Varken.py
@@ -1,33 +1,36 @@
 import platform
 import schedule
+import distro
 from time import sleep
 from queue import Queue
 from sys import version
 from threading import Thread
 from os import environ as env
 from os import access, R_OK, getenv
-from distro import linux_distribution
 from os.path import isdir, abspath, dirname, join
 from argparse import ArgumentParser, RawTextHelpFormatter
 from logging import getLogger, StreamHandler, Formatter, DEBUG
 
+
 # Needed to check version of python
 from varken import structures  # noqa
 from varken.ombi import OmbiAPI
+from varken.overseerr import OverseerrAPI
 from varken.unifi import UniFiAPI
 from varken import VERSION, BRANCH, BUILD_DATE
 from varken.sonarr import SonarrAPI
 from varken.radarr import RadarrAPI
 from varken.lidarr import LidarrAPI
 from varken.iniparser import INIParser
 from varken.dbmanager import DBManager
+from varken.influxdb2manager import InfluxDB2Manager
 from varken.helpers import GeoIPHandler
 from varken.tautulli import TautulliAPI
 from varken.sickchill import SickChillAPI
 from varken.varkenlogger import VarkenLogger
 
 
-PLATFORM_LINUX_DISTRO = ' '.join(x for x in linux_distribution() if x)
+PLATFORM_LINUX_DISTRO = ' '.join(distro.id() + distro.version() + distro.name())
 
 
 def thread(job, **kwargs):
@@ -90,7 +93,15 @@ def thread(job, **kwargs):
     vl.logger.info("Varken v%s-%s %s", VERSION, BRANCH, BUILD_DATE)
 
     CONFIG = INIParser(DATA_FOLDER)
-    DBMANAGER = DBManager(CONFIG.influx_server)
+
+    if CONFIG.influx2_enabled:
+        # Use INFLUX version 2
+        vl.logger.info('Using INFLUXDBv2')
+        DBMANAGER = InfluxDB2Manager(CONFIG.influx_server)
+    else:
+        vl.logger.info('Using INFLUXDB')
+        DBMANAGER = DBManager(CONFIG.influx_server)
+
     QUEUE = Queue()
 
     if CONFIG.sonarr_enabled:
@@ -156,6 +167,18 @@ def thread(job, **kwargs):
                 at_time = schedule.every(server.issue_status_run_seconds).seconds
                 at_time.do(thread, OMBI.get_issue_counts).tag("ombi-{}-get_issue_counts".format(server.id))
 
+    if CONFIG.overseerr_enabled:
+        for server in CONFIG.overseerr_servers:
+            OVERSEER = OverseerrAPI(server, DBMANAGER)
+            if server.get_request_total_counts:
+                at_time = schedule.every(server.request_total_run_seconds).seconds
+                at_time.do(thread, OVERSEER.get_request_counts).tag("overseerr-{}-get_request_counts"
+                                                                    .format(server.id))
+            if server.num_latest_requests_to_fetch > 0:
+                at_time = schedule.every(server.num_latest_requests_seconds).seconds
+                at_time.do(thread, OVERSEER.get_latest_requests).tag("overseerr-{}-get_latest_requests"
+                                                                     .format(server.id))
+
     if CONFIG.sickchill_enabled:
         for server in CONFIG.sickchill_servers:
             SICKCHILL = SickChillAPI(server, DBMANAGER)
@@ -171,7 +194,8 @@ def thread(job, **kwargs):
 
     # Run all on startup
     SERVICES_ENABLED = [CONFIG.ombi_enabled, CONFIG.radarr_enabled, CONFIG.tautulli_enabled, CONFIG.unifi_enabled,
-                        CONFIG.sonarr_enabled, CONFIG.sickchill_enabled, CONFIG.lidarr_enabled]
+                        CONFIG.sonarr_enabled, CONFIG.sickchill_enabled, CONFIG.lidarr_enabled,
+                        CONFIG.overseerr_enabled]
     if not [enabled for enabled in SERVICES_ENABLED if enabled]:
         vl.logger.error("All services disabled. Exiting")
         exit(1)

diff --git a/data/varken.example.ini b/data/varken.example.ini
@@ -3,10 +3,12 @@ sonarr_server_ids = 1,2
 radarr_server_ids = 1,2
 lidarr_server_ids = false
 tautulli_server_ids = 1
-ombi_server_ids = 1
+ombi_server_ids = false
+overseerr_server_ids = 1
 sickchill_server_ids = false
 unifi_server_ids = false
 maxmind_license_key = xxxxxxxxxxxxxxxx
+influx2_enabled = false
 
 [influxdb]
 url = influxdb.domain.tld
@@ -16,6 +18,15 @@ verify_ssl = false
 username = root
 password = root
 
+[influx2]
+url = influxdb2.domain.tld
+org = ORG
+token = TOKEN
+timeout = 10000
+ssl = false
+verify_ssl = false
+bucket = varken
+
 [tautulli-1]
 url = tautulli.domain.tld:8181
 fallback_ip = 1.1.1.1
@@ -95,6 +106,17 @@ request_total_run_seconds = 300
 get_issue_status_counts = true
 issue_status_run_seconds = 300
 
+[overseerr-1]
+url = overseerr.domain.tld
+apikey = xxxxxxxxxxxxxxxx
+ssl = false
+verify_ssl = false
+get_request_total_counts = true
+request_total_run_seconds = 30
+get_latest_requests = true
+num_latest_requests_to_fetch = 10
+num_latest_requests_seconds = 30
+
 [sickchill-1]
 url = sickchill.domain.tld:8081
 apikey = xxxxxxxxxxxxxxxx

diff --git a/docker-compose.yml b/docker-compose.yml
@@ -6,7 +6,7 @@ services:
   influxdb:
     hostname: influxdb
     container_name: influxdb
-    image: influxdb
+    image: influxdb:1.8
     networks:
       - internal
     volumes:
@@ -22,91 +22,6 @@ services:
       - /path/to/docker-varken/config-folder:/config
     environment:
       - TZ=America/Chicago
-      - VRKN_GLOBAL_SONARR_SERVER_IDS=1,2
-      - VRKN_GLOBAL_RADARR_SERVER_IDS=1,2
-      - VRKN_GLOBAL_LIDARR_SERVER_IDS=false
-      - VRKN_GLOBAL_TAUTULLI_SERVER_IDS=1
-      - VRKN_GLOBAL_OMBI_SERVER_IDS=1
-      - VRKN_GLOBAL_SICKCHILL_SERVER_IDS=false
-      - VRKN_GLOBAL_UNIFI_SERVER_IDS=false
-      - VRKN_GLOBAL_MAXMIND_LICENSE_KEY=xxxxxxxxxxxxxxxx
-      - VRKN_INFLUXDB_URL=influxdb.domain.tld
-      - VRKN_INFLUXDB_PORT=8086
-      - VRKN_INFLUXDB_SSL=false
-      - VRKN_INFLUXDB_VERIFY_SSL=false
-      - VRKN_INFLUXDB_USERNAME=root
-      - VRKN_INFLUXDB_PASSWORD=root
-      - VRKN_TAUTULLI_1_URL=tautulli.domain.tld:8181
-      - VRKN_TAUTULLI_1_FALLBACK_IP=1.1.1.1
-      - VRKN_TAUTULLI_1_APIKEY=xxxxxxxxxxxxxxxx
-      - VRKN_TAUTULLI_1_SSL=false
-      - VRKN_TAUTULLI_1_VERIFY_SSL=false
-      - VRKN_TAUTULLI_1_GET_ACTIVITY=true
-      - VRKN_TAUTULLI_1_GET_ACTIVITY_RUN_SECONDS=30
-      - VRKN_TAUTULLI_1_GET_STATS=true
-      - VRKN_TAUTULLI_1_GET_STATS_RUN_SECONDS=3600
-      - VRKN_SONARR_1_URL=sonarr1.domain.tld:8989
-      - VRKN_SONARR_1_APIKEY=xxxxxxxxxxxxxxxx
-      - VRKN_SONARR_1_SSL=false
-      - VRKN_SONARR_1_VERIFY_SSL=false
-      - VRKN_SONARR_1_MISSING_DAYS=7
-      - VRKN_SONARR_1_MISSING_DAYS_RUN_SECONDS=300
-      - VRKN_SONARR_1_FUTURE_DAYS=1
-      - VRKN_SONARR_1_FUTURE_DAYS_RUN_SECONDS=300
-      - VRKN_SONARR_1_QUEUE=true
-      - VRKN_SONARR_1_QUEUE_RUN_SECONDS=300
-      - VRKN_SONARR_2_URL=sonarr2.domain.tld:8989
-      - VRKN_SONARR_2_APIKEY=yyyyyyyyyyyyyyyy
-      - VRKN_SONARR_2_SSL=false
-      - VRKN_SONARR_2_VERIFY_SSL=false
-      - VRKN_SONARR_2_MISSING_DAYS=7
-      - VRKN_SONARR_2_MISSING_DAYS_RUN_SECONDS=300
-      - VRKN_SONARR_2_FUTURE_DAYS=1
-      - VRKN_SONARR_2_FUTURE_DAYS_RUN_SECONDS=300
-      - VRKN_SONARR_2_QUEUE=true
-      - VRKN_SONARR_2_QUEUE_RUN_SECONDS=300
-      - VRKN_RADARR_1_URL=radarr1.domain.tld
-      - VRKN_RADARR_1_APIKEY=xxxxxxxxxxxxxxxx
-      - VRKN_RADARR_1_SSL=false
-      - VRKN_RADARR_1_VERIFY_SSL=false
-      - VRKN_RADARR_1_QUEUE=true
-      - VRKN_RADARR_1_QUEUE_RUN_SECONDS=300
-      - VRKN_RADARR_1_GET_MISSING=true
-      - VRKN_RADARR_1_GET_MISSING_RUN_SECONDS=300
-      - VRKN_RADARR_2_URL=radarr2.domain.tld
-      - VRKN_RADARR_2_APIKEY=yyyyyyyyyyyyyyyy
-      - VRKN_RADARR_2_SSL=false
-      - VRKN_RADARR_2_VERIFY_SSL=false
-      - VRKN_RADARR_2_QUEUE=true
-      - VRKN_RADARR_2_QUEUE_RUN_SECONDS=300
-      - VRKN_RADARR_2_GET_MISSING=true
-      - VRKN_RADARR_2_GET_MISSING_RUN_SECONDS=300
-      - VRKN_LIDARR_1_URL=lidarr1.domain.tld:8686
-      - VRKN_LIDARR_1_APIKEY=xxxxxxxxxxxxxxxx
-      - VRKN_LIDARR_1_SSL=false
-      - VRKN_LIDARR_1_VERIFY_SSL=false
-      - VRKN_LIDARR_1_MISSING_DAYS=30
-      - VRKN_LIDARR_1_MISSING_DAYS_RUN_SECONDS=300
-      - VRKN_LIDARR_1_FUTURE_DAYS=30
-      - VRKN_LIDARR_1_FUTURE_DAYS_RUN_SECONDS=300
-      - VRKN_LIDARR_1_QUEUE=true
-      - VRKN_LIDARR_1_QUEUE_RUN_SECONDS=300
-      - VRKN_OMBI_1_URL=ombi.domain.tld
-      - VRKN_OMBI_1_APIKEY=xxxxxxxxxxxxxxxx
-      - VRKN_OMBI_1_SSL=false
-      - VRKN_OMBI_1_VERIFY_SSL=false
-      - VRKN_OMBI_1_GET_REQUEST_TYPE_COUNTS=true
-      - VRKN_OMBI_1_REQUEST_TYPE_RUN_SECONDS=300
-      - VRKN_OMBI_1_GET_REQUEST_TOTAL_COUNTS=true
-      - VRKN_OMBI_1_REQUEST_TOTAL_RUN_SECONDS=300
-      - VRKN_OMBI_1_GET_ISSUE_STATUS_COUNTS=true
-      - VRKN_OMBI_1_ISSUE_STATUS_RUN_SECONDS=300
-      - VRKN_SICKCHILL_1_URL=sickchill.domain.tld:8081
-      - VRKN_SICKCHILL_1_APIKEY=xxxxxxxxxxxxxxxx
-      - VRKN_SICKCHILL_1_SSL=false
-      - VRKN_SICKCHILL_1_VERIFY_SSL=false
-      - VRKN_SICKCHILL_1_GET_MISSING=true
-      - VRKN_SICKCHILL_1_GET_MISSING_RUN_SECONDS=300
     depends_on:
       - influxdb
     restart: unless-stopped
@@ -118,7 +33,7 @@ services:
       - internal
     ports:
       - 3000:3000
-    volumes: 
+    volumes:
       - /path/to/docker-grafana/config-folder:/config
     environment:
       - GF_PATHS_DATA=/config/data
@@ -128,4 +43,4 @@ services:
     depends_on:
       - influxdb
       - varken
-    restart: unless-stopped
+    restart: unless-stopped
diff --git a/requirements.txt b/requirements.txt
@@ -2,9 +2,10 @@
 # Potential requirements.
 # pip3 install -r requirements.txt
 #---------------------------------------------------------
-requests==2.21
+requests==2.28.1
 geoip2==2.9.0
 influxdb==5.2.0
-schedule==0.6.0
+schedule==1.1.0
 distro==1.4.0
-urllib3==1.24.2
+urllib3==1.26.10
+influxdb-client==1.14.0
diff --git a/utilities/historical_tautulli_import.py b/utilities/historical_tautulli_import.py
@@ -41,7 +41,7 @@
     DBMANAGER = DBManager(CONFIG.influx_server)
 
     if CONFIG.tautulli_enabled:
-        GEOIPHANDLER = GeoIPHandler(DATA_FOLDER)
+        GEOIPHANDLER = GeoIPHandler(DATA_FOLDER, CONFIG.tautulli_servers[0].maxmind_license_key)
         for server in CONFIG.tautulli_servers:
             TAUTULLI = TautulliAPI(server, DBMANAGER, GEOIPHANDLER)
             TAUTULLI.get_historical(days=opts.days)
diff --git a/varken.xml b/varken.xml
@@ -51,5 +51,6 @@
   <Labels/>
   <Config Name="PGID" Target="PGID" Default="" Mode="" Description="Container Variable: PGID" Type="Variable" Display="always" Required="true" Mask="false">99</Config>
   <Config Name="PUID" Target="PUID" Default="" Mode="" Description="Container Variable: PUID" Type="Variable" Display="always" Required="true" Mask="false">100</Config>
+  <Config Name="Debug" Target="DEBUG" Default="False" Mode="" Description="Turn Debug on or off" Type="Variable" Display="always" Required="false" Mask="false">False</Config>
   <Config Name="Varken DataDir" Target="/config" Default="" Mode="rw" Description="Container Path: /config" Type="Path" Display="advanced-hide" Required="true" Mask="false">/mnt/user/appdata/varken</Config>
 </Container>
diff --git a/varken/influxdb2manager.py b/varken/influxdb2manager.py
@@ -0,0 +1,48 @@
+from sys import exit
+from logging import getLogger
+import influxdb_client
+from influxdb_client import InfluxDBClient
+from influxdb_client.client.write_api import SYNCHRONOUS
+
+
+class InfluxDB2Manager(object):
+    def __init__(self, server):
+        self.server = server
+        self.logger = getLogger()
+        if self.server.url == "influxdb2.domain.tld":
+            self.logger.critical("You have not configured your varken.ini. Please read Wiki page for configuration")
+            exit()
+
+        self.influx = InfluxDBClient(url=self.server.url, token=self.server.token, org=self.server.org,
+                                     timeout=self.server.timeout, verify_ssl=self.server.verify_ssl,
+                                     ssl_ca_cert=self.server.ssl)
+        self.influx_write_api = self.influx.write_api(write_options=SYNCHRONOUS)
+
+        # Create the bucket if needed
+
+        bucket_api = self.influx.buckets_api()
+
+        try:
+            bucket = bucket_api.find_bucket_by_name(self.server.bucket)
+
+            if bucket is None:
+                self.logger.info('Creating bucket %s', self.server.bucket)
+
+                org_api = influxdb_client.service.organizations_service.OrganizationsService(self.influx.api_client)
+                orgs = org_api.get_orgs()
+                for org in orgs.orgs:
+                    if org.name == self.server.org:
+                        my_org = org
+
+                self.influx.buckets_api().create_bucket(bucket_name=self.server.bucket, org_id=my_org.id)
+        except Exception as e:
+            self.logger.error('Failed creating new InfluxDB bucket! Error: %s', e)
+
+    def write_points(self, data):
+        self.logger.info('Writing Data to InfluxDBv2 %s', data)
+
+        try:
+            self.influx_write_api.write(bucket=self.server.bucket, record=data)
+        except Exception as e:
+            self.logger.exception('Error writing data to influxdb2. Dropping this set of data. '
+                                  'Check your database! Error: %s', e)