Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot get webserver to run properly #1

Closed
ahuse opened this issue Feb 15, 2022 · 20 comments
Closed

Cannot get webserver to run properly #1

ahuse opened this issue Feb 15, 2022 · 20 comments
Labels
bug Something isn't working

Comments

@ahuse
Copy link

ahuse commented Feb 15, 2022

This tool is awesome, thank you. But i am having a hard setting it up properly.

I got the Web-Interface running, but i cannot add podcasts through it (credentials set, no error message). When i add a podcast through the CLI the podcast shows up on the main page, but the served .index for the podcast is empty.

Can you write a small tutorial how to set up the webserver or point me in the right direction?

I am using a RPi 4 with 64-bit bullseye and nginx, php-fpm.

@ahuse ahuse changed the title Cannot get webserver to run properly Cannot get webserver to run properly label:question Feb 15, 2022
@ahuse ahuse changed the title Cannot get webserver to run properly label:question label:question Cannot get webserver to run properly Feb 15, 2022
@ahuse ahuse changed the title label:question Cannot get webserver to run properly Cannot get webserver to run properly Feb 15, 2022
@Yetangitu
Copy link
Owner

First check if the webserver user - most likely www-data on that RPi - can access the spodcast command, e.g.:

$ sudo -u www-data bash
$ spodcast -h

If that part works the rest should be easy. Since you're using nginx you can just take the snippet from the last section of the README, adjust the server_name and root to your configuration and things should work.

server {
        listen 80;
        listen [::]:80;
        server_name spodcast.example.org;

        root /mnt/audio/spodcast;

        index .index.php;

        # these files should not be accessible
        location ~\.(json|info)$ {
                deny all;
                return 404;
        }

        location / {
                try_files $uri $uri/ =404;
        }

        location ~ \.php$ {
                include snippets/fastcgi-php.conf;
                fastcgi_pass unix:/run/php/php7.4-fpm.sock;
        }
}

So, from start to finish:

  1. download and extract the distribution tarball or clone the repo
  2. cd spodcast && pip install .
  3. test if the web server user has access to spodcast, if not either adjust permissions so it works, run pip install . as root so it gets installed globally or run it as the web server user (www-data on Debian and derived distributions)
  4. add the above nginx configuration snippet to the server configuration, e.g. by creating a file in /etc/nginx/sites-available with this content which you then link to /etc/nginx/sites-enabled. Things may differ on your installation, check how nginx was installed and adjust these locations accordingly.
  5. Test the configuration by attempting to sync a podcast. This should produce a new directory under the --root-path named after the show you synced, containing one or more episodes and the corresponding *.info files. It should also contain an index.info file with information about the show as well as a hidden .index.php file which combines the information in all *.info files into an RSS feed. In --root-path you should find another hidden .index.php file containing the feed manager, a feed.info containing information about all shows, the spodcast configuration file you configured (that would be spodcast.json (from -c /mnt/audio/spodcast/spodcast.json) for the above example) and a settings.info which contains information about updates and the site url. Some of these configuration files only appear once you have started to use the web UI, on a fresh install you should find at least the root .index.php, spotcast,json and spotcast-cred-......json (assuming you used the -l spotify.rc command).

Have a look at the contents of your --root-path, does it contain all needed bits?

@ahuse
Copy link
Author

ahuse commented Feb 15, 2022

Ok. Thanks for the quick reply.

I installed globally via sudo pip install and had to edit the root path in /mnt/audio/spodcast/spodcast.json.

I can now create feeds and the corresponding folders are created. Files are downloaded too. But when i click on the feed link, it just load an empty .html. index.info and .index.phpare there and not empty. Permissions should be ok.

The link is http://rpi4/The_Joe_Rogan_Experience/. Maybe a problem because this is not a toplevel domain? Or is something wrong with php-fpm?

As you may noticed, i am not an expert in web-dev (😅), but maybe you have an idea what is going on here.

@Yetangitu
Copy link
Owner

Shows can not be top-level domains since they always have the manager below them so that should not be the problem. It sounds more as if the PHP code in .index.php is not executed on load, is there anything interesting in the php-fpm or nginx error logs?
Does the feed manager work as it should? It should look somewhat like the screenshots and allow you to add/delete/configure/refresh shows and set global update parameters (in the Settings screen).

@ahuse
Copy link
Author

ahuse commented Feb 15, 2022

Feed manager looks normal. I can add/delete/configure/refresh shows and set parameters.

Here is the error log from nginx while i am trying to access the show.

2022/02/15 17:30:02 [error] 717#717: *37 FastCGI sent in stderr: "PHP message: PHP Parse error: syntax error, unexpected '=' in /mnt/audio/spodcast/The_Joe_Rogan_Experience/.index.php on line 2" while reading response header from upstream, client: 10.153.200.237, server: spodcast.rpi4.local, request: "GET /The_Joe_Rogan_Experience/ HTTP/1.1", upstream: "fastcgi://unix:/run/php/php7.4-fpm.sock:", host: "rpi4", referrer: "http://rpi4/"

@Yetangitu
Copy link
Owner

OK, I see the problem - it lies in the template for the feed .index.php , I'll post a new version RSN which should work out of the box. Hold on...

@Yetangitu
Copy link
Owner

https://github.com/Yetangitu/Spodcast/releases/tag/v0.3.2

The only significant change is to the index feed template which was missing two const keywords. Either re-install this new version or edit the template yourself:

--- a/spodcast/feedgenerator.py
+++ b/spodcast/feedgenerator.py
@@@ -4,8 -4,8 +4,8 @@@ RSS_FEED_SHOW_INDEX = 'index
  
  def RSS_FEED_CODE():
      return r'''<?php
--SHOW_INDEX = "''' + RSS_FEED_SHOW_INDEX + r'''";
--INFO = "''' + RSS_FEED_INFO_EXTENSION + r'''";
++const SHOW_INDEX = "''' + RSS_FEED_SHOW_INDEX + r'''";
++const INFO = "''' + RSS_FEED_INFO_EXTENSION + r'''";
  header("Content-type: text/xml");
  $feed_name = "Spodcast autofeed";
  $feed_description = "Spodcast autofeed";

@ahuse
Copy link
Author

ahuse commented Feb 15, 2022

Now it is working, kind off...

The rss-feed is loading fine, but each episode is served as a https://... link. When i change the base-url in .index.php to http://... i can open and play the episodes in the (terrible) iOS podcast-app.

Overcast, the podcast app i normally use, gives me an error on adding the podcast. At the moment i do not know why, but i will try to investigate.

Thanks for you help so far.

@Yetangitu
Copy link
Owner

I ll change that hard-coded https to follow the protocol in the next release. As to why Overcast doesn't like the feed, no idea, maybe it tries to interpret it as an atom feed due to the presence of a single atom:link tag? Try to find a feed which Overcast does accept, look at the structure of <channel> and <item>s in this feed and fiddle with the template in .index.php until it works, then let me know what you had to change?

@Yetangitu
Copy link
Owner

Now fixed in master, if you want to apply the fix yourself:

--- a/spodcast/feedgenerator.py
+++ b/spodcast/feedgenerator.py
@@ -6,10 +6,11 @@ def RSS_FEED_CODE():
     return r'''<?php
 const SHOW_INDEX = "''' + RSS_FEED_SHOW_INDEX + r'''";
 const INFO = "''' + RSS_FEED_INFO_EXTENSION + r'''";
+$PROTOCOL + ($HTTPS) ? "https://" : "http://";
 header("Content-type: text/xml");
 $feed_name = "Spodcast autofeed";
 $feed_description = "Spodcast autofeed";
-$base_url = strtok('https://' . $_SERVER['HTTP_HOST'] . $_SERVER['REQUEST_URI'], '?');
+$base_url = strtok($PROTOCOL . $_SERVER['HTTP_HOST'] . $_SERVER['REQUEST_URI'], '?');
 $feed_logo = "$base_url/.image.jpg";
 $feed_link = $base_url;
 $allowed_extensions = array('mp4','m4a','aac','mp3','ogg');

@ahuse
Copy link
Author

ahuse commented Feb 16, 2022

Ok, i had no luck in changing the template.

I think Overcast can not access the local network due to iOS security. I can not grant local access unless the app asks me to. I dig into that and maybe contact the developer.

Currently i have no VPS to test this suspicion, but maybe you or somebody else has a non-local instance to do a short test.

@Yetangitu
Copy link
Owner

Just install 0.3.3, it works without needing to meddle with the template.

As to testing on iOS that'd be hard for me given the lack of apples other than those from trees. I can rig up a temporary instance for you to test if that helps. Send me some contact details on s p o d c a s t followed by the at symbol followed by unter n e t, dot, org and I'll reply with a temporary domain. Do your testing and let me know if it works. I use letsencrypt certificates, these should work more or less everywhere. I'll remove the domain when you're done testing.

@john-griffin
Copy link

Adding a deploy to Heroku button https://devcenter.heroku.com/articles/heroku-button or a Dockerfile would be super helpful.

@ahuse
Copy link
Author

ahuse commented Feb 16, 2022

Here are my findings so far:

your testing instance works in overcast.

So i did some more testing on my local instance and got it (kind of) working:

I accessed the Pi via its local IP (not hostname), then copied the source-code of a feed to an .xml file and posted this file on my github. Then i added the show via the direct link of the file on github to overcast. With this method overcast asked me to grant local access and downloaded the episodes from the local Pi.

Not working when:

  • i copy the .xml in the local show folder and try to access it with overcast --> timed out warning in overcast
  • the hostname (for example rpi4) is in the .xml and not the actual local IP --> not finding episodes

When overcast is timing out i do not see a hit in the nginx logs.

So maybe as a workaround i have to write a script that generates the .xmls and puts them on github. Not ideal though...

@Yetangitu
Copy link
Owner

Yetangitu commented Feb 16, 2022

[edit: irrelevant, see below for the real solution]

@Yetangitu
Copy link
Owner

Yetangitu commented Feb 16, 2022

This is how the traffic related to Overcast looks on the server, it probably points at your problem. The first Overcast-related traffic looks like this:

104.237.136.79 - - [16/Feb/2022:15:28:20 +0100] "GET /Gemischtes_Hack/ HTTP/1.1" 200 4959 "-" "Overcast/1.0 Podcast Sync (0 subscribers; feed-id=0; +http://overcast.fm/)"
104.237.136.79 - - [16/Feb/2022:15:28:21 +0100] "GET /Gemischtes_Hack/ HTTP/1.1" 200 4959 "-" "Overcast/1.0 Podcast Sync (0 subscribers; feed-id=0; +http://overcast.fm/)"
172.104.194.166 - - [16/Feb/2022:15:28:21 +0100] "GET /Gemischtes_Hack/Gemischtes_Hack_-_173_SPAGHETTI_FUR_9.mp3 HTTP/1.1" 301 169 "-" "Overcast/1.0 Podcast Sync (+http://overcast.fm/)"
172.104.194.166 - - [16/Feb/2022:15:28:27 +0100] "GET /Gemischtes_Hack/Gemischtes_Hack_-_173_SPAGHETTI_FUR_9.mp3 HTTP/2.0" 200 104598771 "-" "Overcast/1.0 Podcast Sync (+http://overcast.fm/)"

Notice that the traffic does not come from your device, it comes from the Overcast mothership:

$ host 104.237.136.79
79.136.237.104.in-addr.arpa domain name pointer crawl5.overcast.fm.

Only after crawling the feed from the central authority does Overcast-the-app come into action:

xxx.xxx.xxx.xxx - - [16/Feb/2022:15:28:34 +0100] "GET /Gemischtes_Hack/Gemischtes_Hack_-_173_SPAGHETTI_FUR_9.mp3 HTTP/1.1" 301 169 "-" "Overcast/3.0 (+http://overcast.fm/; iOS podcast app)"
xxx.xxx.xxx.xxx - - [16/Feb/2022:15:28:49 +0100] "GET /Gemischtes_Hack/Gemischtes_Hack_-_173_SPAGHET9888888888888888888888888888888888TI_FUR_9.mp3 HTTP/2.0" 200 104598771 "-" "Overcast/3.0 (+http://overcast.fm/; iOS podcast app)"

...followed by Overcast central, again:

104.237.137.166 - - [16/Feb/2022:15:30:46 +0100] "GET /Nur_verheiratet_mit_Hazel__Thomas_ HTTP/1.1" 301 169 "-" "Overcast/1.0 Podcast Sync (0 subscribers; feed-id=0; +http://overcast.fm/)"
104.237.137.166 - - [16/Feb/2022:15:30:47 +0100] "GET /Nur_verheiratet_mit_Hazel__Thomas_/ HTTP/1.1" 200 6864 "https://4c3bb03d.example.org/Nur_verheiratet_mit_Hazel__Thomas_" "Overcast/1.0 Podcast Sync (0 subscribers; feed-id=0; +http://overcast.fm/)"
104.237.136.161 - - [16/Feb/2022:15:30:47 +0100] "GET /Nur_verheiratet_mit_Hazel__Thomas_ HTTP/1.1" 301 169 "-" "Overcast/1.0 Podcast Sync (0 subscribers; feed-id=0; +http://overcast.fm/)"                                                   
104.237.136.161 - - [16/Feb/2022:15:30:48 +0100] "GET /Nur_verheiratet_mit_Hazel__Thomas_/ HTTP/1.1" 200 6864 "https://4c3bb03d.example.org/Nur_verheiratet_mit_Hazel__Thomas_" "Overcast/1.0 Podcast Sync (0 subscribers; feed-id=0; +http://overcast.fm/)"                                                                                                                                                                                                                                  
192.53.162.67 - - [16/Feb/2022:15:30:48 +0100] "GET /Nur_verheiratet_mit_Hazel__Thomas_/Nur_verheiratet_mit_Hazel__Thomas__-_Wir_feiern_Hochzeitstag_mit_Gasten_VOL._1.ogg HTTP/1.1" 301 169 "-" "Overcast/1.0 Podcast Sync (+http://overcast.fm/)"                                                                                                                                                                                                                                           
192.53.162.67 - - [16/Feb/2022:15:30:52 +0100] "GET /Nur_verheiratet_mit_Hazel__Thomas_/Nur_verheiratet_mit_Hazel__Thomas__-_Wir_feiern_Hochzeitstag_mit_Gasten_VOL._1.ogg HTTP/2.0" 200 52577177 "-" "Overcast/1.0 Podcast Sync (+http://overcast.fm/)"                                                                                                                                                                                                                                      

...upon which the app on your device seems to be given permission by the Overlords to access the feed:

xxx.xxx.xxx.xxx - - [16/Feb/2022:15:30:58 +0100] "GET /Nur_verheiratet_mit_Hazel__Thomas_/Nur_verheiratet_mit_Hazel__Thomas__-_Wir_feiern_Hochzeitstag_mit_Gasten_VOL._1.ogg HTTP/1.1" 301 169 "-" "Overcast/3.0 (+http://overcast.fm/; iOS podcast app)"                                                                                                                                                                                                                                     
etc,                                                                                                                                                                                                                                           

In other words, that Overcast app does not directly accept podcast links like, say, Antennapod on Android or just about any other real podcast app does. It submits "foreign" podcast links to some centralised service which then tells the app what to do with them in some way.
The real solution here is to use a real podcast app, one which does not rely on any other servers than those which serve the feed. The next-best solution is to make your Pi accessible from the outside to Overcast. Given the nature of these feeds - content from Spotify - I think this is not an optimal solution and likely to lead to problems if Spodcast or something like it were to take off.

To paraphrase some old greybeard: here's a nickel, son. Get yourself a real podcast app.

@Yetangitu
Copy link
Owner

FYI, Spodcast is now available on PyPi, just run pip install spodcast to install it for personal use, sudo pip install spodcast to install it globally.

@Yetangitu
Copy link
Owner

Adding a deploy to Heroku button https://devcenter.heroku.com/articles/heroku-button or a Dockerfile would be super helpful.

Feel like submitting a PR for one of those? I don't use either so I'd have to create a build environment for Docker and get involved with Heroku. Running my own services on my own hardware on my own connections this is not that relevant for me.

@ahuse
Copy link
Author

ahuse commented Feb 16, 2022

Thanks for providing the logs. Seems like you got to the root.

As Overcast is an awesome app i just wrote a script, which uploads the .xml to github and runs as a cron. This should work for me.

@Yetangitu
Copy link
Owner

...until Overcast.inc starts to wonder what that strange feed coming from github is. I have no idea whether they police the feeds gathered by its users so with a bit of luck this will keep on working.

@Yetangitu
Copy link
Owner

I'll close this issue seeing as how Spodcast works as intended.

@Yetangitu Yetangitu added the bug Something isn't working label Feb 17, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants