Ability to select a proxy

[Bilal2453]

yt-dlp through its --proxy CLI option can accept a variety of protocols for proxying the connection. While it is possible to set the proxy from GNOME settings (and expect it would work, although frankly only https_proxy worked from me testing this app), this is sadly not the case for isolated environments such as Flatpak.
For those you will have to set the environment variable (https_proxy/all_proxy) manually when launching the app from CLI.

An option in the app settings to set a proxy should make this easier.

Perhaps, it could be interesting to see something like "Pass additional flags to yt-dlp" option, although not sure how possible is this.

[nlogozzo]

Perhaps, it could be interesting to see something like "Pass additional flags to yt-dlp" option, although not sure how possible is this.

Parabolic doesn't call yt-dlp via CLI but instead actually generates and calls Python code using the yt-dlp python API, so we wouldn't be able to have a "pass additional flags"....but we can certainly implement proxy settings as an individual preference

[Bilal2453]

but instead actually generates and calls Python code using the yt-dlp python API

That was indeed by concern here, oh well.

[nlogozzo]

https://github.com/NickvisionApps/Parabolic/releases/tag/2023.8.0-rc1

[Bilal2453]

@nlogozzo Could you please re-confirm that this is working? And if it does work how did you test it?

I am not only unable to get this to work, but also getting behavior I am not exactly expecting!

Setting the proxy preference to any URL (that is not a real proxy server) would still let the app operate just fine (which is confusing if it was really using the proxy). I tried accessing blocked websites (in my region) using a general HTTP proxy (with the value http://IP_ADDRESS:PORT) and using the Tor network (with the value socks5://127.0.0.1:9050) and both options don't seem to affect the request in any way. The blocked site is still inaccessible, and sites (such as YouTube) that block Tor access by default still work. Indicating that this is not working as expected, they do work as expected following the steps I described initially. Same applies to GNOME settings.

Here is the troubleshooting report:

rg.nickvision.tubeconverter
2023.8.0-rc1
GTK 4.10.4
libadwaita 1.3.3
Flatpak
yt-dlp 2023.07.06
psutil 5.9.5
ffmpeg version 6.0
aria2 version 1.36.0
en-US
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

[Bilal2453]

Also I have some concerns relating #491, does the request that gets sent to check for internet connection use the set proxy? It should if it currently does not, as in some specific applications, it can be that the internet access is only possible through that proxy. For example, when using non-root WiFi relays on an Android system.

[fsobolev]

Also I have some concerns relating #491, does the request that gets sent to check for internet connection use the set proxy? It should if it currently does not, as in some specific applications, it can be that the internet access is only possible through that proxy. For example, when using non-root WiFi relays on an Android system.

By default the app asks NetworkManager for network status without sending any requests itself. Only if this fails (for example, if running on a system without NetworkManager), then it resorts to pinging multiple reliable addresses to detect whether there's full internet connection.

[Bilal2453]

Right, that time when NetworkManager reports a failure, but a proxy is set up in the app's preference, is what I am talking about here.

[Bilal2453]

Oh I see what you actually meant with "fails". Hmm, not much can be done here I guess as it would complicate this quite a bit. For example browsers can re-route their DNS queries over the HTTP proxy (when DNS-over-HTTPS is used), I don't think that is really feasible here sadly.

[nlogozzo]

@nlogozzo Could you please re-confirm that this is working? And if it does work how did you test it?
I am not only unable to get this to work, but also getting behavior I am not exactly expecting!

Sorry my fault!! Should be fixed with this release: https://github.com/NickvisionApps/Parabolic/releases/tag/2023.8.0-rc2
Will be available through flathub-beta within the hour :)

[nlogozzo]

Right, that time when NetworkManager reports a failure, but a proxy is set up in the app's preference, is what I am talking about here.

Well if you have a proxy set up on your system (via GNOME Control Panel or whatever), that's what NetworkManager will use to test your connection.

Basically, if you can access the internet on your system (one way or another), then NetworkManager can detect the connection.

If NetworkManager "fails", which is only likely on a non-linux system where there is no NetworkManager, then that list that @fsobolev shared of URLs will be pinged to test connection.

[Bilal2453]

Sorry my fault!

No worries! happens to the best of us.

Won't be able to test it within an hour, may reply again later this day to confirm. Although it looks good code wise so no need for that confirmation.

[nlogozzo]

Sorry my fault!

No worries! happens to the best of us.

Won't be able to test it within an hour, may reply again later this day to confirm. Although it looks good code wise so no need for that confirmation.

Yeah I forget to actually set the url to the provided string and just left it "" which is why it wasnt working 😆

But no rush, whenever you get a chance :)

[Bilal2453]

Hm, something is not quite right. Still observing the same behavior on 2023.8.0-rc2.

[fsobolev]

Just tried 2023.8.0-rc2. I don't have a sample video that I could only download only with proxy and also I don't know a proxy that would work for sure, so I tried to download random youtube video using random free proxy. With multiple proxies that I tried I didn't manage to download video, was getting errors (mostly timed out error), but as soon as I disabled proxy it downloaded as usual, so the setting seems to work. At least for downloading videos, but not for validating URLs, this should be fixed.

[nlogozzo]

Hm, something is not quite right. Still observing the same behavior on 2023.8.0-rc2.

As Fyodor said, the validation wasn't accepting the proxy hence why it wasn't working (i don't know how we missed all this 😆)

Should be all good now in the new update: https://github.com/NickvisionApps/Parabolic/releases/tag/2023.8.0-rc3

[Bilal2453]

Oh yep, that looks like it, I should have specified this was on validation specifically. This is now behaving as I would expect.

Thank you, and keep the good work up!

[nlogozzo]

Glad it's working!! Stable release planned for Friday :)

[Bilal2453]

By the way, it doesn't seem to me like it is respecting GNOME's settings at all.

Setting this proxy in the app's preference, I can access blocked site, but not when the app's preference is empty and GNOME's is set instead. Is this perhaps to do with Flatpak's environment isolation? Not really that important for me although.

(I tried setting SOCKS field only first, then set HTTPS field too then HTTP, after each trying to validation the URL and that fails)

Note: it is possible to simulate region block websites by blocking a domain on the firewall side, or the home router side.

[nlogozzo]

By the way, it doesn't seem to me like it is respecting GNOME's settings at all.

Parabolic doesn't look at GNOME's proxy settings so that's expected.
Parabolic is a yt-dlp frontend, so it allows you to specify the proxy url in Parabolic preferences to pass to yt-dlp via its --proxy flag.

Does yt-dlp from the command line respect the proxy without specifying the --proxy flag? If not, then it's a request for yt-dlp.

[Bilal2453]

Does yt-dlp from the command line respect the proxy without specifying the --proxy flag? If not, then it's a request for yt-dlp.

Yes, it is a somewhat established behavior: if an app can accept a proxy, it should check the environment variables for one.

This is also how I was able to get Parabolic to use a proxy before it supported that, setting an environment variables for the parent process will let yt-dlp recognize that and use it (It is actually the underlying Python library if I am not mistaken).

The way GNOME's Setting works is that when you set an HTTP(s) proxy, an environment variables (http_proxy and https_proxy) is set to that value, if an FTP proxy is set, it sets the environment variable ftp_proxy, if a socks5 proxy (or other) is set it uses the environment variable all_proxy. Those environment variables are then passed to each and every program (user launched processes).

I assume this is not working because Flatpak isolates the system's environment variables from those of the app, you will need someway of letting Flatpak pass those environment variables if they are set.. or just ignore this all together perhaps.