"Wget --spider" not working through local proxy, breaks Puppy Package Manager

[ghost]

1 CONNECT TO WIFI

(I am connecting through a WiFi Direct connection from my mobile phone.
This is an ad-hoc network, I do not know if this could be the cause of my issue, but it is the reason I need the proxy to connect...)

2 SET PROXY

/usr/local/simple_network_setup/proxy-setup
(My proxy is http://192.168.49.1:8000, I set this for http & ftp, no user/pass)
Reboot.

3 OPEN PUPPY PACKAGE MANAGER & UPDATE

Clicked Configure package manager->Update database->Update Now
...Update downloads package lists & pops up a dialog saying it has completed.
(NOTE: This did not work until I rebooted, because my proxy settings had not been set on bootup yet...)

4 SEARCH FOR PROXYCHAINS & TRY TO INSTALL

5 SEARCH FOR DELUGE & TRY TO INSTALL

Found a little info here, but nothing else in that thread worked either. Have there been any significant changes to how puppy package manager connects to the repos? I took a quick look at a few scripts to see if I could find a ping or something that was not making it through the proxy, but I didn't find anything...

[wdlkmpx]

I can't say much about the ppm, i rarely use it. But you might want to try this enhanced proxy-setup script and see if it makes any difference..

It has nice gui improvements, more error checking, exports more environment variables so that more apps are aware of the proxy settings, etc.

proxy-setup.zip

[ghost]

Appreciate the improved script, but ppm is sticking to it's guns. I ran your script, rebooted, same error. I don't understand, because my web browsers work through the proxy. And if I remove all proxy settings and connect to a network that doesn't use a proxy, ppm works fine. It's a pain to look for a proper connection where I am at, so I guess I can only install packages when I go out some place that has a better net connection... :/

[mavrothal]

Does PPM work at all? ie does it work with other simple packages, like a desktop theme pet?
Which puppy do you have the problem with? Is it with any puppy (if you use more that one)?

This is strange as both database update (0setup script) and package download (downlaod_file script) are using wget to download the respective files. Is interesting that in one case works with the proxy and in the other does not. downlaod_file is using the spider mode so this might be an issue.

If indeed database update works with proxy and PPM can still not find ANY package with proxy, then you can tell wget to use a proxy with the -e option ie in line 67 of /usr/sbin/download_file use LANG=C wget -e use_proxy=on -e http_proxy=192.168.49.1:8000 -4 -t 2 -T 20 --waitretry=20 --spider -S "${URLSPEC}" > /tmp/download_file_spider.log1 2>&1
(if it fails try use_proxy=yes as older wget version use this)

If PPM finds some packages and fails in others then is the database that needs updating or the remote server has issues. If you use bionic this is happening these days.

[mavrothal]

Regarding the proxy command in wget, you may also want to modify accordingly lines 219, 224 and 227 of /usr/local/petget/downloadpkgs.sh

[ghost]

IT IS A PING!
Why is ppm pinging an address to determine if I can download something if it's going to use wget for the download?
Any way I can stop ppm from sending that ping out?
Because ping doesn't seem to work behind proxies...
https://superuser.com/questions/175428/how-to-ping-when-behind-a-proxy/175441#175441

[ghost]

Seems a few people have had this same issue with pupget a long time ago and were able to circumvent the ping test with a hacked .pup of petget, but that does me no good since it is so old.
Here is a thread of someone detailing their frustration over the same issue...

Here is the console output up until it fails at "testing whether packages exist in repository..."
Before I have to open my browser and download them from the same urls that ppm fails to...

root# ppm EXIT=" Ok " /usr/bin/gcc ping: bad address 'distro.ibiblio.org' ping: bad address 'distro.ibiblio.org' /usr/local/petget/installpreview.sh: line 180: 17928 Terminated . /usr/lib/gtkdialog/box_splash -close never -text "$(gettext 'Please wait, processing package database files...')" /usr/local/petget/downloadpkgs.sh: line 240: 18480 Terminated . yaf-splash -bg '#FFD600' -close never -fontsize large -text "$(gettext 'Please wait, testing that packages exist in repository...')" /usr/local/petget/ui_Classic: line 206: 17014 Killed . /usr/lib/gtkdialog/box_splash -close never -text "$(gettext 'Loading Puppy Package Manager...')"

[mavrothal]

PPM pings to checks if there is a connection but this does not stop it from working.
As you see this is not a "common" issue with puppy, so would be nice if you could help providing some more info to address the issue.
Can you try to see if the suggested changes above helped and give some info for your set up?
How the proxy was set up, so we can automate its detection?. What is the output of echo $http_proxy and cat /var/local/proxy_server for example.

[ghost]

Ok, that's good to hear. That thread had me thinking I had no recourse. The proxy is setup by an app called PDAnet on my phone. THIS is the setup tutorial they sent to me.

I tried the changes you proposed before, but they seemed to have no effect. I tried them with "use_proxy=on" and "use_proxy=yes" per your suggestion. Tried to install FCEUX, same deal. Errors at testing whether packages exist.

root# echo $http_proxy
http://192.168.49.1:8000
root# cat /var/local/proxy_server
cat: /var/local/proxy_server: No such file or directory
^COULD THAT BE AN ISSUE?^

[ghost]

So this is quite interesting! wget does not work in spider mode, but does work normally... How do I fix that???

[mavrothal]

Actually in your spider cal you do not have a correct URL (distro.ibiblio.org)
But to test that what is the output of

wget -4 -t 2 -T 20 --waitretry=20 --spider -S http://mirrors.kernel.org/ubuntu/pool/main/b/bolt/bolt_0.2-0ubuntu1_i386.deb

# and then

wget -e use_proxy=on -e http_proxy=192.168.49.1:8000 -4 -t 2 -T 20 --waitretry=20 --spider -S http://mirrors.kernel.org/ubuntu/pool/main/b/bolt/bolt_0.2-0ubuntu1_i386.deb

?
(copy/paste if possible)
Thx

[ghost]

I did notice i screwed up the url on the second command, so I'm going to fix the picture I uploaded, as it is an extremely poor demonstration of what I was attempting to convey....
anyways...

root# wget -e use_proxy=on -e http_proxy=192.168.49.1:8000 -4 -t 2 -T 20 --waitretry=20 --spider -S http://mirrors.kernel.org/ubuntu/pool/main/b/bolt/bolt_0.2-0ubuntu1_i386.deb
Spider mode enabled. Check if remote file exists.
--2018-04-20 08:36:40-- http://mirrors.kernel.org/ubuntu/pool/main/b/bolt/bolt_0.2-0ubuntu1_i386.deb
Connecting to 192.168.49.1:8000... connected.
Proxy request sent, awaiting response... Read error (Connection reset by peer) in headers.
Retrying.

Spider mode enabled. Check if remote file exists.
--2018-04-20 08:36:41-- (try: 2) http://mirrors.kernel.org/ubuntu/pool/main/b/bolt/bolt_0.2-0ubuntu1_i386.deb
Connecting to 192.168.49.1:8000... connected.
Proxy request sent, awaiting response... Read error (Connection reset by peer) in headers.
Giving up.

root# wget -4 -t 2 -T 20 --waitretry=20 --spider -S http://mirrors.kernel.org/ubuntu/pool/main/b/bolt/bolt_0.2-0ubuntu1_i386.deb
Spider mode enabled. Check if remote file exists.
--2018-04-20 08:38:31-- http://mirrors.kernel.org/ubuntu/pool/main/b/bolt/bolt_0.2-0ubuntu1_i386.deb
Connecting to 192.168.49.1:8000... connected.
Proxy request sent, awaiting response... Read error (Connection reset by peer) in headers.
Retrying.

Spider mode enabled. Check if remote file exists.
--2018-04-20 08:38:32-- (try: 2) http://mirrors.kernel.org/ubuntu/pool/main/b/bolt/bolt_0.2-0ubuntu1_i386.deb
Connecting to 192.168.49.1:8000... connected.
Proxy request sent, awaiting response... Read error (Connection reset by peer) in headers.
Giving up.

[ghost]

Set these values in /etc/wgetrc:
# You can set the default proxies for Wget to use for http, https, and ftp.
# They will override the value in the environment.
https_proxy = http://192.168.49.1:8000/
http_proxy = http://192.168.49.1:8000/
ftp_proxy = http://192.168.49.1:8000/
`# If you do not want to use proxy at all, set this to off.`
use_proxy = on

no changes detected so far... :/
is this a bug in wget?
any workaround?

[mavrothal]

The error suggest that might be some issues with your proxy configuration probably the port forwarding or blocking of web crawlers (ie spider mode) by your proxy.
Just in case though try dropping "-4" that enforce IPv4 only. ie wget -t 2 -T 20 --waitretry=20 --spider -S .....

[ghost]

dropping the -4 argument produces the same results...
Read error (Connection reset by peer) in headers.
As the proxy for the connection is created by PDAnet, and not myself, I have no control over it's settings...

Honestly, I think it's more likely the wget --spider command is trying to circumvent the proxy settings.

Couldn't the check for the file be done with a curl script like this [source]?
url=$"http://distro.ibiblio.org/puppylinux/Packages-puppy-xenial64-official"
if curl --output /dev/null --silent --fail -r 0-0 "$url"; then
echo "URL exists: $url"
else
echo "URL does not exist: $url"
fi
because running this script gives me this:
URL exists: http://distro.ibiblio.org/puppylinux/Packages-puppy-xenial64-official

How feasible is this to implement in your opinion?

[mavrothal]

Curl is more powerful (and bigger) than wget, but is usually is not part of puppy.
A temp way to circumvent the problem is comment out the exit commands (line 244 in downloadpkgs.sh and line 74 in download_file) till a proper solution is found.
The problem here is that If the file does not exist you end up with half-done installations.

[mavrothal]

... a couple more options in case they can bypass your pesky proxy
Try these 2 commands alone or in combination in wget
--span-hosts and -e robots=off
also try to fake wget as a browser with something like
--user-agent="Mozilla/5.0 (X11; Ubuntu; Linux x86; rv:52.0) Gecko/20100101 Firefox/52.0"

[ghost]

same errors with:
1.) --span-hosts
2.) -e robots=off
3.) --span-hosts -e robots=off
4.) --user-agent="Mozilla/5.0 (X11; Ubuntu; Linux x86; rv:52.0) Gecko/20100101 Firefox/52.0"

So I tried commenting the two lines you mentioned, but I still receive an error saying it cannot determine the filesizes but they do seem to exist.

This is starting to feel like beating a dead horse. This has become a lot of effort for naught. I just wanted to use the package manager that came with my distro.

[mavrothal]

OK. Thanks for you input.
You might want to contact your proxy administrator as wget spider mode should work behind proxies.
Regarding using PPM under these conditions you could also comment out lines 117 and 165 of download_file but then you add more potential troubles, specially if you run with a limited size savefile. So I think is not a good idea.
Also you may want to be more specific with your title to facilitate potential fixing. Something like "wget spider mode fails behind a proxy, breaking Puppy Package Manager"

[wdlkmpx]

I remember editing the ppm to fix some things i found annoying. all those changes got lost (in early 2016 i think) when i hesitated to add the changes.

I guess this is not an issue in ./1download (build system), as it tries to download packages without checking in spider mode first. Except the arm SD skeleton image, code i didnt touch.

wget should just attempt to download the file without checking in spider mode first.

This is the kind of stuff that just should work even with proxies... they call it superiority.

And it's a privilege to have someone who can provide all the required evidence to fix this.

He who has ears to hear, let him hear.

[mavrothal]

wget should just attempt to download the file without checking in spider mode first.

Given the way puppy is built and run, checking for presence and size is important so you do not end up with broken builds or a full savefile (people still use it) or be prompted to update your dbs. Of course for an experience linux user these are not an issue but this is not necessarily the puppy base.

On the other hand PPM could check hashes at startup with remote site to make sure package db are up to date and make sure both that connections are alive and the packages exist, bypassing the spider check. What we need here is hashes for the ibiblio pet databases and a some code in 3builddistro (or 0setup) to download and record the original db hashes.
(One concern here is the "complains" about not explicitly authorised connections and downloads 🙄 but an authorisation could be requested on every run, configurable in settings)

I'm sure there are other solutions (including removing all checks) but before we get to this or a similar exercise should point out that wget could/should work fine with a properly configured proxy.

[wdlkmpx]

In 1download it's easy to tell when a package has been downloaded correctly or not downloaded at all [ -s ]... the script extracts the contents to test. When phil had problems downloading uget and could not figure out why it failed, i could easily tell it was a corrupted package by looking at the screen and how the script kept telling the pkg was corrupted and kept downloading it from all the mirrors... The problem is that it doesn't do anything obvious to alert you if you don't see it on time, i'll add red letters and a pause. Or could it just stop processing to let you know that something must be done whether you want it or not... The same behavior can be applied to the ppm. The ppm needs too many fixes... but i think a new cli package manager (probably written in C) must be written from scratch and a new primitive ppm should be frontend. One time the spider mode was actually making 1download slow as hell for me, i don't know why but the spider took like 10 seconds to finish its task before the actual download happened.. and i had to dowload 400+ packages. Maybe i should also add a md5sum, or sha256sum.. what should be used now?

[ghost]

Ok, so to anyone reading this, just to be clear, if you are experiencing this very specific issue, commenting out lines 74, 117, & 165 of /usr/sbin/download_file and line 244 in /usr/local/petget/downloadpkgs.sh still gave all the same errors (and new ones!), but it continued to download and install the packages. It's ugly, it's a bad method, I know, but it works. Thank you for the help. I appreciate it immensely.

I'm glad we could narrow down the cause a bit. I was going to try to get in touch with the maintainers of wget for a possible solution, but apparently you can only report bugs through a mailing list-thing. Didn't feel worth the effort.

As I said, the proxy is setup by PDAnet App on my phone. I don't know why you immediately want to say it's misconfigured. Wget spider mode does definitely work through an external proxy, but I cannot say I have ever seen it work through a local one. It may be possible that spider mode uses ICMP protocol to communicate. Proxy only supports TCP and UDP packets. This is the problem with Ping.

[wdlkmpx]

Maybe downloadpkgs should not run wget in spider mode, as it calls download_file, which runs wget in spider mode again, i'll take a look

[wdlkmpx]

This is not related to this issue, but i ended up noticing it..

In installpkg.sh, there is post-processing that only takes in frugal mode, but as far i know it also affects full installs, when you install a package containing DISTRO_ARCHDIR, it overwrites the symlink no matter the pupmode, there's a workaround to avoid that situation lines above it

'120102 install may have overwritten a symlink-to-dir... ' this can also happen in a full install, quite convoluted code here. The script shouldn't attempt to fix every possible situation with incompatible or unsuitable packages..

There's a huge code block i deleted in 2createpackages, in a pull request.. broken code, regarding fixing desktop file icon and category. The same code i see in installpkg.sh... I tested with 30-40 packages containing broken icons paths or something like that, it failed all my tests.

To fix this, we should add a fairly complete hicolor icon theme, i added more IconPaths to jwm_config, allowing it to find many more icons (this fixed quite a few of the missing icons in my build)... There are also broken icon paths for some old pet packages, that requires fixing those packages

In short, there's code fix and delete..