Split api and config logic

[RynoM]

Hi,

Really like the idea for this project and totally see a use for this myself. I've been playing with it a bit for fun and made some changes that might be nice. Hope this is useful. Thoughts/changes:

• split api logic and configuration (applying) logic
• easily extendible Config class, possible other data formats to load from
• moved api_key to the config, as part of 'server' (needs to be in the config anyways for prowlarr, and parsing initialize.js seems a bit hacky?)
• added some constants/enum/types

I dont have a good setup yet to test many settings for multiple of the *arr applications though, but functionality shouldn't have been changed.

[PierreMesure]

Hi Ryno!

Thanks a lot, I love the refactor!

I think in the long run, I’d like to keep using the initialize.js. I have no idea why they make it so easy to retrieve the API key but I don’t find it hacky, it makes it more seamless to the user and reduces the chance that API key ends up in a committed config file.

Now, I agree with you that it gets hacky when the API keys are provided in clear in the Prowlarr config. In a future update, I’d like to make Flemmarr fetch the URLs and API keys for applications automatically.

[PierreMesure]

In the Dockerfile, I split the ADD for caching purpose. When adding requirements.txt alone at the beginning, it doesn’t need to fetch these requirements every time it builds the image. But it might slightly increase the size of the image to have this one additional step. I’ll check and merge if that’s the case.

[RynoM]

Fair point about the api key, parsing the javascript seems a bit fragile (but probably won't change), but I think its nice if the api key could be specified as a backup. So I added both :). I think having to specify api keys for prowlarr in plaintext is maybe a showstopper though, so I added some support for environment variables.

Also split the dockerfile again based on your comment, but moved the requirements.txt to root where I think it belongs.

[PierreMesure]

Hi @RynoM, I'd love to merge this PR now that I'm back from holiday. I'll try to review it and merge it later this week.

[RynoM]

@PierreMesure let me know if you need anything

[PierreMesure]

Hi @RynoM! Sorry, I got back from holidays and got sucked in daily work (and some other new projects). I won't have time to look at it this weekend but maybe next one. How are you progressing with exporting an existing config?

[RynoM]

Hey @PierreMesure , this time I was the one on holiday. Back now and might continue on it this weekend. When I left it, it was pretty close to a working PoC for 1-2 of the services. How do you want to go about this PR in relation to the new feature of exporting current configs?

[RynoM]

@PierreMesure Exporting current configs seems pretty stable (I think) in this other branch: https://github.com/RynoM/Flemmarr/tree/feature/get-current-config
Would love to get thoughts/feedback on it. It still needs some more docstrings and comments, but hope the intent is clear.

[gottsman]

@PierreMesure Exporting current configs seems pretty stable (I think) in this other branch: https://github.com/RynoM/Flemmarr/tree/feature/get-current-config Would love to get thoughts/feedback on it. It still needs some more docstrings and comments, but hope the intent is clear.

@RynoM Tried out you repo and to get a sample dump from brand new local *arr instances. I removed the config.yml to try to force a backup dump. I am struggling with specifying my API keys in this setup. I am getting the following error

❯ docker logs -f flemmarr
Initializing connection to lidarr
No api key in config, fetching api key instead.

[RynoM]

@RynoM Tried out you repo and to get a sample dump from brand new local *arr instances. I removed the config.yml to try to force a backup dump. I am struggling with specifying my API keys in this setup. I am getting the following error

❯ docker logs -f flemmarr
Initializing connection to lidarr
No api key in config, fetching api key instead.

Hi @gottsman, There is currently no difference between error and info logs, so I get the confusion. The logs you are seeing are from expected behavior. You should be getting a new file called config_backup.yml in your mounted volume (same folder as the config.yml.

The code will try to make a backup first, no matter if a config.yml exists or not, but if one does exist it will apply it after making a backup.

With regards to the api key; I removed the ability to specify one in this branch, since it can be done automagically. However it would be quite easy to allow the explicit specification again if needed/wanted.

Let me know how it goes! :)

[gottsman]

@PierreMesure Exporting current configs seems pretty stable (I think) in this other branch: https://github.com/RynoM/Flemmarr/tree/feature/get-current-config Would love to get thoughts/feedback on it. It still needs some more docstrings and comments, but hope the intent is clear.

The patches from @RynoM work brilliantly and as expected. I built and tested against linux/arm64 and linux/amd64. My previous issues were related to docker networking in my personal setup.

[RynoM]

The patches from @RynoM work brilliantly and as expected. I built and tested against linux/arm64 and linux/amd64. My previous issues were related to docker networking in my personal setup.

Awesome! :) Did you happen to try backup and re-apply or just backing up? Curious if the whole flow works well.

[gottsman]

The patches from @RynoM work brilliantly and as expected. I built and tested against linux/arm64 and linux/amd64. My previous issues were related to docker networking in my personal setup.

Awesome! :) Did you happen to try backup and re-apply or just backing up? Curious if the whole flow works well.

I've not been able to successfully push the config. Probably user error :-) I still am unsure how to specify the API key (either in the compose file or the resulting backup config yml).

I also am getting a warning on my M1 Mac. It doesn't provide the same warning on a linux VM (AMD64). I don't that variable defined in my compose file. Perhaps its a docker system variable.

❯ dc up -d
WARN[0000] The "Jty" variable is not set. Defaulting to a blank string.

[RynoM]

@gottsman You don't have to specify an API key for Flemmarr to be able to connect to your services, it will call the 'initialize' endpoint on each service and retrieve the key from there.

The log you posted doesn't look related I think. Do you have any logs from the Flemmarr container specifically? If something went wrong there should be errors / a stack trace in there.

[gottsman]

@gottsman You don't have to specify an API key for Flemmarr to be able to connect to your services, it will call the 'initialize' endpoint on each service and retrieve the key from there.

The log you posted doesn't look related I think. Do you have any logs from the Flemmarr container specifically? If something went wrong there should be errors / a stack trace in there.

I ran Flemmarr on a test VM with the backup config exported from the family media server. Although I provided the config.yml, it still attempted to create the backup config. The new backup config was roughly smaller than the "real" config. Logs below....

Initializing connection to lidarr
No api key in config, fetching api key instead.
Fetching: http://lidarr:8686/api/v1/health
Successfully connected to the server.
Fetching: http://lidarr:8686/api/v1/customfilter
Fetching: http://lidarr:8686/api/v1/delayprofile
Fetching: http://lidarr:8686/api/v1/downloadclient
Fetching: http://lidarr:8686/api/v1/config/downloadclient
Fetching: http://lidarr:8686/api/v1/config/host
Fetching: http://lidarr:8686/api/v1/importlist
Fetching: http://lidarr:8686/api/v1/importlistexclusion
Fetching: http://lidarr:8686/api/v1/indexer
Fetching: http://lidarr:8686/api/v1/config/indexer
Fetching: http://lidarr:8686/api/v1/config/mediamanagement
Fetching: http://lidarr:8686/api/v1/metadata
Fetching: http://lidarr:8686/api/v1/metadataprofile
Fetching: http://lidarr:8686/api/v1/config/metadataprovider
Fetching: http://lidarr:8686/api/v1/config/naming
Fetching: http://lidarr:8686/api/v1/notification
Fetching: http://lidarr:8686/api/v1/qualitydefinition
Fetching: http://lidarr:8686/api/v1/qualityprofile
Fetching: http://lidarr:8686/api/v1/releaseprofile
Fetching: http://lidarr:8686/api/v1/remotepathmapping
Fetching: http://lidarr:8686/api/v1/rootfolder
Fetching: http://lidarr:8686/api/v1/tag
Fetching: http://lidarr:8686/api/v1/tag/detail
Fetching: http://lidarr:8686/api/v1/config/ui
Initializing connection to radarr
No api key in config, fetching api key instead.
Fetching: http://radarr:7878/api/v3/health
Successfully connected to the server.
Fetching: http://radarr:7878/api/v3/customfilter
Fetching: http://radarr:7878/api/v3/customformat
Fetching: http://radarr:7878/api/v3/delayprofile
Fetching: http://radarr:7878/api/v3/downloadclient
Fetching: http://radarr:7878/api/v3/config/downloadclient
Fetching: http://radarr:7878/api/v3/config/host
Fetching: http://radarr:7878/api/v3/exclusions
Fetching: http://radarr:7878/api/v3/config/importlist
Fetching: http://radarr:7878/api/v3/indexer
Fetching: http://radarr:7878/api/v3/config/indexer
Fetching: http://radarr:7878/api/v3/config/mediamanagement
Fetching: http://radarr:7878/api/v3/metadata
Fetching: http://radarr:7878/api/v3/config/metadata
Fetching: http://radarr:7878/api/v3/config/naming
Fetching: http://radarr:7878/api/v3/notification
Fetching: http://radarr:7878/api/v3/qualitydefinition
Fetching: http://radarr:7878/api/v3/qualityprofile
Fetching: http://radarr:7878/api/v3/remotepathmapping
Fetching: http://radarr:7878/api/v3/restriction
Fetching: http://radarr:7878/api/v3/rootfolder
Fetching: http://radarr:7878/api/v3/tag
Fetching: http://radarr:7878/api/v3/tag/detail
Fetching: http://radarr:7878/api/v3/config/ui
Initializing connection to prowlarr
No api key in config, fetching api key instead.
Fetching: http://prowlarr:9696/api/v1/health
Successfully connected to the server.
Fetching: http://prowlarr:9696/api/v1/applications
Fetching: http://prowlarr:9696/api/v1/appprofile
Fetching: http://prowlarr:9696/api/v1/customfilter
Fetching: http://prowlarr:9696/api/v1/config/development
Fetching: http://prowlarr:9696/api/v1/downloadclient
Fetching: http://prowlarr:9696/api/v1/config/downloadclient
Fetching: http://prowlarr:9696/api/v1/config/host
Fetching: http://prowlarr:9696/api/v1/indexer
Fetching: http://prowlarr:9696/api/v1/indexerproxy
Fetching: http://prowlarr:9696/api/v1/notification
Fetching: http://prowlarr:9696/api/v1/tag
Fetching: http://prowlarr:9696/api/v1/tag/detail
Fetching: http://prowlarr:9696/api/v1/config/ui
Initializing connection to sonarr
No api key in config, fetching api key instead.
Fetching: http://sonarr:8989/api/v3/health
Successfully connected to the server.
Fetching: http://sonarr:8989/api/v3/customfilter
Fetching: http://sonarr:8989/api/v3/delayprofile
Fetching: http://sonarr:8989/api/v3/downloadclient
Fetching: http://sonarr:8989/api/v3/config/downloadclient
Fetching: http://sonarr:8989/api/v3/config/host
Fetching: http://sonarr:8989/api/v3/importlist
Fetching: http://sonarr:8989/api/v3/importlistexclusion
Fetching: http://sonarr:8989/api/v3/indexer
Fetching: http://sonarr:8989/api/v3/config/indexer
Fetching: http://sonarr:8989/api/v3/languageprofile
Fetching: http://sonarr:8989/api/v3/config/mediamanagement
Fetching: http://sonarr:8989/api/v3/metadata
Fetching: http://sonarr:8989/api/v3/config/naming
Fetching: http://sonarr:8989/api/v3/notification
Fetching: http://sonarr:8989/api/v3/qualitydefinition
Fetching: http://sonarr:8989/api/v3/qualityprofile
Fetching: http://sonarr:8989/api/v3/releaseprofile
Fetching: http://sonarr:8989/api/v3/remotepathmapping
Fetching: http://sonarr:8989/api/v3/rootfolder
Fetching: http://sonarr:8989/api/v3/tag
Fetching: http://sonarr:8989/api/v3/tag/detail
Fetching: http://sonarr:8989/api/v3/config/ui
Backed-up current configurations
Initializing connection to lidarr
No api key in config, fetching api key instead.
Fetching: http://lidarr:8686/api/v1/health
Successfully connected to the server.
Fetching: http://lidarr:8686/api/v1/config/ui/1
Fetching: http://lidarr:8686/api/v1/customfilter
Fetching: http://lidarr:8686/api/v1/delayprofile/1
Fetching: http://lidarr:8686/api/v1/delayprofile
Fetching: http://lidarr:8686/api/v1/downloadclient/1
Traceback (most recent call last):
  File "/api.py", line 68, in update
    current_settings = self.get(resource, id)
  File "/api.py", line 51, in get
    response.raise_for_status()
  File "/usr/local/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://lidarr:8686/api/v1/downloadclient/1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "//run.py", line 20, in <module>
    main()
  File "//run.py", line 15, in main
    cfg.apply(params)
  File "/config.py", line 47, in apply
    self._triage_and_apply(self[app], api)
  File "/config.py", line 59, in _triage_and_apply
    self._triage_and_apply(obj[key], api, f"{resource}/{key}")
  File "/config.py", line 64, in _triage_and_apply
    api.update(resource, item, id)
  File "/api.py", line 73, in update
    return self.create(resource, body)
  File "/api.py", line 57, in create
    response.raise_for_status()
  File "/usr/local/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http://lidarr:8686/api/v1/downloadclient

[RynoM]

@gottsman The way I have it setup now, it creates a backup every time, regardless of if you provide a config. Thanks for the logs though, it helped me identify an issue I hadn't spotted yet. Unfortunately I don't have much time at the moment, but I will try to work on a solution.