Some content not working when behind Traefik reverse proxy #239
Comments
|
Here's a list of the podcast feeds which I've found issues with Looking at the XML for the top podcast as an example <rss xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd" version="2.0">
<channel>
<title>99% Invisible</title>
<link>https://podfetch.SERVER.COM/rss/21</link>
<description>Design is everywhere in our lives, perhaps most importantly in the places where we've just stopped noticing. 99% Invisible is a weekly exploration of the process and power of design and architecture. From award winning producer Roman Mars. Learn more at 99percentinvisible.org.</description>
<language>en-us</language>
<category>Arts</category>
<itunes:author>Roman Mars</itunes:author>
<itunes:category text="Arts"/>
<itunes:image href="https://podfetch.SERVER.COM/podcasts/99% Invisible/image.jpg"/>
<itunes:explicit>false</itunes:explicit>
<itunes:new-feed-url>https://podfetch.SERVER.COM/rss/21</itunes:new-feed-url>
<itunes:owner>
<itunes:name>Roman Mars</itunes:name>
<itunes:email>local@local.com</itunes:email>
</itunes:owner>
<itunes:summary>Design is everywhere in our lives, perhaps most importantly in the places where we've just stopped noticing. 99% Invisible is a weekly exploration of the process and power of design and architecture. From award winning producer Roman Mars. Learn more at 99percentinvisible.org.</itunes:summary>
<itunes:keywords>Arts</itunes:keywords>If I try to access Note the URL above is actually changed to Now for the next podcast "Ask Noah" <rss xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd" version="2.0">
<channel>
<title>Ask Noah Show</title>
<link>https://podfetch.SERVER.COM/rss/5</link>
<description>The Ask Noah Show is a weekly talk radio show where we focus on Linux and Open Source technology. We invite the community to participate live on the air 1-855-450-6624. The show airs Tuesdays at 6pm CT on asknoahshow.com and at KEQQ 88.3 FM in Grand Forks ND. It's a free call 1-855-450-NOAH so join us and start on your way to owning your operating system, your software, and technology.</description>
<language>en-us</language>
<category>Technology</category>
<category>News</category>
<category>Education</category>
<itunes:author>Noah J. Chelliah</itunes:author>
<itunes:category text="Technology"/>
<itunes:category text="News"/>
<itunes:category text="Education"/>
<itunes:image href="https://podfetch.SERVER.COM/podcasts/Ask Noah Show/image.jpg"/>
<itunes:explicit>false</itunes:explicit>
<itunes:new-feed-url>https://podfetch.SERVER.COM/rss/5</itunes:new-feed-url>
<itunes:owner>
<itunes:name>Noah J. Chelliah</itunes:name>
<itunes:email>local@local.com</itunes:email>
</itunes:owner>
<itunes:summary>The Ask Noah Show is a weekly talk radio show where we focus on Linux and Open Source technology. We invite the community to participate live on the air 1-855-450-6624. The show airs Tuesdays at 6pm CT on asknoahshow.com and at KEQQ 88.3 FM in Grand Forks ND. It's a free call 1-855-450-NOAH so join us and start on your way to owning your operating system, your software, and technology.</itunes:summary>
<itunes:keywords>Technology,News,Education</itunes:keywords>Using the URL In summary I'm not sure this is a singular issue, but more of a general thing where resolving thumbnails/downloading thumbails fails. |
|
Have continued digging into this, managed to fix the issue with no thumbnails for the podcast by manually replacing On the "Latest Episodes" however the thumbnails still aren't displayed presumedly as all the The podcast "99% Invisible" however I think is a completely different issue. Now, interestingly enough, this podcast broke Mazda car stereos (if I remember correctly it was due to the The podcast feed is added without error, but it's not possible to play a lot of the podcast episodes. |
|
As you can see it works without a problem. Can you replicate the issue and give me a step by step guide? |
|
@SamTV12345 That's interesting, yeah I can definitely replicate it. Have nuked and paved a few times now to try and iron out issues. EDIT: As for the step by step guide, it's essentially the Here's a log from when I've just added 99% Invisible using the iTunes podcast search facility tried to play the first episode. The icon still doesn't appear for me. |
|
Also would you mind trying one of the other podcast feeds and seeing if you can replicate the lack of icons appearing? |
|
OK, getting somewhere now. Removing Authelia & Traefik from the equation resolves the issues. Happy to try and work to get this resolved though, as I suspect as podfetch gains traction I'm not going to be the first person to hit this issue. |
I tried it on a separate server and it did work. You also get some errors in the logs. |
2023-07-27T11:56:23.255034343Z thread 'tokio-runtime-worker' panicked at 'failed to copy content: Custom { kind: Other, error: reqwest::Error { kind: Body, source: hyper::Error(Body, Custom { kind: UnexpectedEof, error: IncompleteBody }) } }', src/service/download_service.rs:90:47 |
|
It's definitely something to do with Traefik. It all works as expected until I put it behind the Traefik reverse proxy. So changing my to fixes all the issues. My gut feeling is it's something to do with websockets, but I'm struggling to know where to go next with this. |
|
So your setup is Traefik -> PodFetch? |
If you set it to http PodFetch won't establish a secure websocket leading to mixed content which most browsers will block. |
Yes, and Traefik is exposing only port 443 for SSL |
|
And you have Traefik set to route that traefik to http://podfetch-ip? Could you share your traefik config anonymized? |
Using http://192.168.0.250:8000 everything works correctly, so if I'm correct in my understanding when I put Traefik in front my broswer is then blocking anything on the websocket which is served on http. In that case, is there anyway to force podfetch is serve on https? The alternative is to see if I can force a websocket http to https upgrade. |
Yeah, I'll sanitise my config and post it here, gimme a few minutes. |
Ah that's the problem. podfetch doesn't know anything about SSL. The SERVER_URL is the url that you type into the browser. Try http and maybe another port. I'm just curious how you can do that config in Traefik as I want to move from Nginx Proxy Manger to Traefik as I added a DNS-0 challenge for my preferred Dyn DNS provider. |
|
global:
checkNewVersion: true
sendAnonymousUsage: false
serversTransport:
insecureSkipVerify: true
api:
dashboard: true
insecure: false
entryPoints:
https:
address: ":443"
proxyProtocol:
trustedIPs:
- 192.168.0.0/16
forwardedHeaders:
trustedIPs:
- 127.0.0.1/32
- 192.168.0.0/16
http:
tls:
certResolver: letsencrypt-dns
domains:
- main: 'server.com'
sans:
- '*.server.com'
providers:
docker:
endpoint: "tcp://docker-socket-proxy:2375"
exposedByDefault: false
defaultRule: "Host(`{{ index .Labels \"com.docker.compose.service\"}}.server.com`)"
network: traefik
file:
directory: "/configs"
watch: true
certificatesResolvers:
letsencrypt-dns:
acme:
email: myemail@email.com
storage: acme.json
#caServer: "https://acme-staging-v02.api.letsencrypt.org/directory" #Uncomment for staging
dnsChallenge:
provider: cloudflare
delayBeforeCheck: 20s
resolvers:
- "1.1.1.1:53"
- "1.0.0.1:53"
disablePropagationCheck: true
accessLog:
filePath: "/logs/access.log"
fields:
headers:
names:
User-Agent: keep
filters:
statusCodes:
- "400-499"
retryAttempts: true
# By default, the level is set to ERROR, Set to DEBUG, PANIC, FATAL, ERROR, WARN, and INFO.
log:
level: ERROR
http:
middlewares:
authelia:
forwardauth:
address: http://authelia:9091/api/verify?rd=https://login.{{env "TRAEFIK_DOMAIN"}}/ # replace auth with your authelia container name
trustForwardHeader: true
authResponseHeaders:
- Remote-User
- Remote-Groups
- Remote-Name
- Remote-Email
http:
middlewares:
securityHeaders:
headers:
customResponseHeaders:
X-Robots-Tag: "none,noarchive,nosnippet,notranslate,noimageindex"
X-Forwarded-Proto: "https"
server: ""
customRequestHeaders:
X-Forwarded-Proto: "https"
sslProxyHeaders:
X-Forwarded-Proto: "https"
referrerPolicy: "same-origin"
hostsProxyHeaders:
- "X-Forwarded-Host"
contentTypeNosniff: true
browserXssFilter: true
forceSTSHeader: true
stsIncludeSubdomains: true
stsSeconds: 63072000
stsPreload: trueThen to run podfetch behind the Traefik reverse proxy I just run this compose file: podfetch:
image: samuel19982/podfetch:postgres
container_name: podfetch
networks:
- podfetch
- traefik
user: 99:100
environment:
- POLLING_INTERVAL=${POLLING_INTERVAL}
- SERVER_URL=${SERVER_URL}
# - GPODDER_INTEGRATION_ENABLED=${GPODDER_INTEGRATION_ENABLED}
- PODINDEX_API_KEY=${PODINDEX_API_KEY}
- PODINDEX_API_SECRET=${PODINDEX_API_SECRET}
- DATABASE_URL=${DATABASE_URL}
- DB_CONNECTIONS=${DB_CONNECTIONS}
volumes:
- ${PODCASTS}/new:/app/podcasts
- ${CONFIG}/podfetch:/app/db
# ports:
# - 8000:8000
restart: always
labels:
- "traefik.enable=true"
- "traefik.http.routers.podfetch.middlewares=authelia@file, securityHeaders@file"
podfetch-postgres:
image: postgres
container_name: podfetch-postgres
networks:
- podfetch
environment:
- POSTGRES_DB=${PODFETCH_POSTGRES_DB}
- POSTGRES_USER=${PODFETCH_POSTGRES_USER}
- POSTGRES_PASSWORD=${PODFETCH_POSTGRES_PASSWORD}
volumes:
- ${CONFIG}/podfetch-postgres:/var/lib/postgresql/data
# ports:
# - 5432:5432
restart: unless-stoppedWith this |
|
Thanks for the detailed configuration. I'll try that. |
|
I can't change my external domain to As regards Traefik, I've never used NPM but have used Nginx for years, I migrated to Traefik a while ago and never looked back I love the fact I just need to add some docker labels to a service to reverse proxy it. If you have an email I can ping you across a guide I wrote for a friend to get him up and running. |
|
Thanks you can contact me via samelus1998@outlook.de . I meant that the internal traefik inside the container network should be http. I'll try to replicate that scenario. Maybe I can find a configuration error. Or maybe try to go from the ip of the host and the port of the container rather than the docker network. |
Ah, gotcha, yeah it already is by default. If I have a container that communicates on |
|
@SamTV12345 Have added you as a collaborator on my private repo. If anything isn't clear then feel free to open an issue. |
arcoast
changed the title
|
So have tried exposing port 8000 and applyng a file based config with Traefik using both: and podfetch:
image: samuel19982/podfetch:postgres
container_name: podfetch
networks:
- podfetch
- traefik
user: 99:100
environment:
- POLLING_INTERVAL=${POLLING_INTERVAL}
- SERVER_URL=${SERVER_URL}
# - GPODDER_INTEGRATION_ENABLED=${GPODDER_INTEGRATION_ENABLED}
- PODINDEX_API_KEY=${PODINDEX_API_KEY}
- PODINDEX_API_SECRET=${PODINDEX_API_SECRET}
- DATABASE_URL=${DATABASE_URL}
- DB_CONNECTIONS=${DB_CONNECTIONS}
volumes:
- ${PODCASTS}:/app/podcasts
- ${CONFIG}/podfetch:/app/db
ports:
- 8000:8000
restart: always
# labels:
# - "traefik.enable=true"
# - "traefik.http.routers.podfetch.middlewares=authelia@file, podfetch@file, securityHeaders@file"
# - "traefik.http.routers.podfetch.middlewares=authelia@file"
# - "traefik.http.routers.podfetch.middlewares=securityHeaders@file"Traefik dynamic file based config http:
routers:
podfetch:
entryPoints:
- https
rule: 'Host(`podfetch.{{env "TRAEFIK_DOMAIN"}}`)'
service: podfetch
middlewares:
- "authelia@file"
- "securityHeaders@file"
- "podfetch@file"
# - "authentik@file"
services:
podfetch:
loadBalancer:
servers:
- url: http://192.168.0.250:8000And tried applying a middleware to the config to upgrade any websocket connections but unfortunately not managed to solve this issue yet. http:
middlewares:
podfetch:
headers:
customRequestHeaders:
X-Forwarded-Proto: "https,wss"My other middlewares for TRaefik are: http:
middlewares:
securityHeaders:
headers:
customResponseHeaders:
X-Robots-Tag: "none,noarchive,nosnippet,notranslate,noimageindex"
X-Forwarded-Proto: "https"
server: ""
customRequestHeaders:
X-Forwarded-Proto: "https"
sslProxyHeaders:
X-Forwarded-Proto: "https"
referrerPolicy: "same-origin"
hostsProxyHeaders:
- "X-Forwarded-Host"
contentTypeNosniff: true
browserXssFilter: true
forceSTSHeader: true
stsIncludeSubdomains: true
stsSeconds: 63072000
stsPreload: trueand http:
middlewares:
authelia:
forwardauth:
address: http://authelia:9091/api/verify?rd=https://login.{{env "TRAEFIK_DOMAIN"}}/ # replace auth with your authelia container name
trustForwardHeader: true
authResponseHeaders:
- Remote-User
- Remote-Groups
- Remote-Name
- Remote-EmailI've tried removing both the securityHeaders & authelia middleware which also doesn't affect the issue. I'm assuming this is an issue trying to serve
I may be completely off piste with all this though as I'm no webdev. |
Thanks for the invite. That looks promising. I'll try to create a Traefik environment at the weekend and create (hopefully) a wiki entry for deploying PodFetch with Traefik. |
Thanks for trying some configs. I hope that https won't be needed at the PodFetch end. I moved PodFetch today to Cloudflare tunnels and that seems to work. So I still have hope that it could be a configuration issue 🤞 . |
|
I'm on holiday for a week so I'll probably not get the chance to do much for a while, but if it's ok with you I'd like to keep trying to get it working behind traefik and I'll post any findings I make so if anyone else comes along with a similar issue they can chip in. |
|
I added a sample configuration here: https://github.com/SamTV12345/PodFetch/tree/feature/terraform/docs/traefik . Next I'll try it with https but so far it works without a problem with http via Traefik. |
|
That link doesn't work, but I took a look at the commit. I suspect it is an issue with mixed http/https content. Interestingly there's another new podcast app and reading their docs, whilst I don't exactly understand it all, it involves a proxy for images/audio files to avoid CORS issues https://www.pinepods.online/docs/intro#proxy-info I wonder if that's the issue here, which may explain why you're finding it's working fine over http. |
I proxy only the audio which can definteley cause issues with cors. I haven't found any cors issues with images. I am currently working on the SSL config in my homelab with Traefik. Maybe then I can find an error. |
|
I added my traefik yaml. |
|
That's the same config as I have albeit in yaml form. Just to help debugging I used the yaml form: http:
routers:
podfetch:
entryPoints:
- https
rule: 'Host(`podfetch.{{env "TRAEFIK_DOMAIN"}}`)'
service: podfetch
services:
podfetch:
loadBalancer:
servers:
- url: http://x.x.x.x:8000Still having the issues I described originally. Struggling to see what I'm missing here. |
|
Ok. I'll try to create a simple docker compose with Traefik and PodFetch tomorrow. Maybe that will work. |
|
I'm getting some browser console errors when trying to access over the reverse proxy. My
and if I run it without the Traefik reverse proxy with the |
Do you have tried it without the slash, because your other URL is also without a slash? |
|
@arcoast Sorry I was busy this weekend. I'll try to reproduce it on Tuesday. |
|
No worries, life gotta come first! |
Hi @helmut72 , yeah I've tried all combinations with no difference in observed behaviour, but thanks for taking a look and making suggestions! |
It should not matter. I always add a slash to the url. |
|
There is a stackoverflow about this http2 error. https://stackoverflow.com/questions/67911570/failed-to-load-resource-neterr-http2-protocol-error-for-react-app-after-upg Did you use multiple platforms/mobile phones? |
|
Or try this dns provider first: https://ipv64.net/account . It is completely free for up to 3 main domains and has unlimited subdomain support. Then you should be able to use my setup. You might need to delete only the cloudflare domain and keep the ipv64 domain. |
Yes I did, tested on Brave browser on both Manjaro KDE and Android and also used Firefox on both platforms too. I'll do some more testing again and see if my wife will let me borrow her Windows machine as I can test on that with Chrome, Edge & Firefox as well. |
Using terraform? Currently I have split DNS so any request from my LAN to I have tried using my 4G/LTE on Android to rule out any wacky issues with DNS. I'd have to learn terraform to try it, I'm not against the idea but it'll take me some time. If you want me to try a docker compose stack that would be much easier. |
|
So, delved into this again and got a friend to test this on his machine to see if it's reproducible and he was able to do so and also provided some really useful insights. I thought given that, it would be useful to post a detailed explanation of the issue we're seeing. First of all, for ease of testing I'm using the SQlite build, but I'm able to reproduce it exactly with postgres too. First of all here's my version: '3.9'
networks:
traefik:
external: false
name: traefik
services:
podfetch:
image: samuel19982/podfetch:latest
container_name: podfetch
networks:
- traefik
user: ${UID:-99}:${GID:-100}
ports:
- 8000:8000
volumes:
- ${CONFIG}/podfetch-sql/podcasts:/app/podcasts
- ${CONFIG}/podfetch-sql/db:/app/db
environment:
- POLLING_INTERVAL=60
- SERVER_URL=https://podfetch.${TRAEFIK_DOMAIN}/
labels:
- "traefik.enable=true"
- "traefik.docker.network=traefik"
- "traefik.http.routers.podfetch.entrypoints=https"
- "traefik.http.routers.podfetch.rule=Host(`podfetch.${TRAEFIK_DOMAIN}`)"
- "traefik.http.services.podfetch.loadbalancer.server.port=8000"My Now once this is installed. Access the podfetch instance at Go to Using The URL of the missing thumbnail is: and trying to load this directly into a browser shows Now reading about URL encoding specifically the line about the Percent character: Which means if you change the above URL which isn't working. to means the thumbnail loads. Now I also noticed that some podcast episodes from 99% Invisible also wouldn't play in the browser which caused me some confusion, but I think I've figured it out. It only seems to be the podcast episodes that have been downloaded. If I attempt to stream a 99% Invisible podcast episode that isn't being cached locally on my server, it plays fine with a URL like this: (These episodes also correctly display the thumbnail too.) However once I download the episode to my server and use However changing this URL by adding in the missing and the MP3 will load correctly. I will need to do some more experimenting, as I had some less significant issues with other podcasts and images, but I haven't looked into it sufficiently yet to work it out. I hope this gives a few more clues on getting to the bottom of things. |
|
So following on from my post yesterday, I've now experimented with running podfetch via Other than So for an image file like above:
And the same phenomenon can be observed for the mp3 files
In summary I think if the URL encoding used in Podfetch is changed to the format in the second type where the |
|
I had a quick look at this as I can replicate in my setup. From: https://github.com/SamTV12345/PodFetch/blob/main/ui/src/utils/Utilities.tsx#L93 It appears that a bespoke URL encoding methodology is employed, cherry picking just two characters which often results in a URL that is a mix of encodied characters and not. This will have unpredictable results depending on the client or proxy and likely explains a lot of these weird symptoms. To avoid edge cases like this and handle internationalisation a URL encoder library should be called instead. |
@nomandera @helmut72 @arcoast I may have found a fix. Could you please try the latest dev images? |
|
Just tried the latest dev image with this commit It's improved things, but not fixed everything with regard to 99% Invisible. The podcast streams are now playing even when cached locally with a URL like this: However the URLs to call |
|
So the images are currently only missing but the podcast plays just fine? |
Yes, that does appear to be the case. |
|
It's a little more subtle that that though, as a browser refresh can cause images that were working to disappear and I think it's because once an image is cached locally it stops working. |
|
Thanks. I'll try to investigate why the images are not working but the podcast url does. They both have the same method applied. |
I just mention the differences, one with one without slahs. But I don't use Traefik. Sorry. |
|
Awesome I could finally replicate the issue. Now I can test that bug. |
That's great, there's been times where I've started to doubt myself here and I've racked my brains trying to work out where I've been going wrong, so that's a relief to hear! Let me know if you want any testing or info and I'm more than happy to help however I can. |
|
@arcoast I think I solved the problem completely. Could you please checkout the dev-urlencoding tag? Try to download all sorts of podcasts and check if you can still play the episodes back. |
|
@SamTV12345 Yep, pulled the 99% Invisible is working perfectly and the thumbnails for other podcasts are now appearing correctly. (I did wonder if that was a separate issue entirely recently, but regardless it seems to be resolved) Can I just say a big "Thank You" for your efforts to resolve this, I know it's been an epic issue in length, it's very much appreciated. I did look the other day for a donation link, but I can't see one anywhere, as I definitely owe you a beer! |
|
You're welcome. That was a lot of work and debug. But I am happy that this is finally solved. I'll merge the branch into main so the image should be there in 10 minutes if my Mac mini doesn't let me down 😄. I unfortunately live in Germany where tax regulations are really strict. I did experiment with GitHub sponsors the other day but it doesn't seem really worth the hassle. So yeah I am also fine with not taking any money. I am always happy if someone gives me a feedback - either positive or negative. So have fun with PodFetch 😃 . |
|
Closed by #263 Confirmed as working with |
Describe the bug
Add a podcast (it doesn't seem to matter which modality used, ie Apple Podcasts, Podindex or my preferred, rss feed) but the icons are not displayed for some podcasts, to clarify, if they work for a podcast series, all the episodes have the thumbnail, if it doesn't work for a podcast series, then none of the episodes have a thumbbail.
I'm installing podfetch with docker-compose, currently on
v4.0.1-16173cd2 (HEAD)My
.envfileThe relevant icon file is indeed present in the folder, it doesn't matter whether the podcast episodes are renamed or not. In the example below, I'm not renaming, but it has no effect either way.
The files have the correct permissions:
It doesn't seem to matter whether I use Firefox or Brave and trying an incognito session doesn't fix the issue.
I've
#ed out the podcasts down below that are unaffected.Nothing is in the logs that I can see either.
Reproduction
Use docker-compose above
System Info
Used Package Manager
npm
Validations
The text was updated successfully, but these errors were encountered: