squid-httpseverywhere module not working

[jaromil]

The new module can be enabled with a symlink in modules/enabled

libxml-libxml-perl must be installed and update-rules run in the same dir.

Then running Dowse start it will generate a perl file with all paths correct to the installation. Squid3 will be launched, but then quit. Here the log:

2014/01/20 21:23:06| WARNING: redirector #1 (FD 9) exited
2014/01/20 21:23:06| WARNING: redirector #2 (FD 11) exited
2014/01/20 21:23:06| WARNING: redirector #3 (FD 13) exited
2014/01/20 21:23:06| WARNING: redirector #4 (FD 15) exited
2014/01/20 21:23:06| Too few redirector processes are running
2014/01/20 21:23:06| storeDirWriteCleanLogs: Starting...
2014/01/20 21:23:06|   Finished.  Wrote 0 entries.
2014/01/20 21:23:06|   Took 0.00 seconds (  0.00 entries/sec).
FATAL: The redirector helpers are crashing too rapidly, need help!

Squid Cache (Version 3.1.20): Terminated abnormally.
CPU Usage: 0.056 seconds = 0.008 user + 0.048 sys
Maximum Resident Size: 28432 KB
Page faults with physical i/o: 0
Memory usage for squid via mallinfo():
        total space in arena:    2248 KB
        Ordinary blocks:         2186 KB     20 blks
        Small blocks:               0 KB      0 blks
        Holding blocks:          2772 KB      5 blks
        Free Small blocks:          0 KB
        Free Ordinary blocks:      61 KB
        Total in use:            4958 KB 221%
        Total free:                61 KB 3%

The generated perl module for squid is found in $DOWSE/run/squid-httpseverywhere.pm

I'm not sure how to run it standalone to have tests, or check its compatibility with Squid3. Needing some help.

[antagonismorg]

What is your redirector_children set to? Try increasing the limit to see if it resolves the issue.

[antagonismorg]

In 2.x, I set url_rewrite_children 40 to resolve the issue.

[jaromil]

It doesn't solves here (stock Debian 7 components, including squid 2)

FATAL: The redirector helpers are crashing too rapidly, need help!

seems the helpers are too few because they crash on launch.

I'm not sure how to debug deeper their problem by running the perl script in a scaffolded way, without squid.

[antagonismorg]

This thread may be helpful. http://marc.info/?l=squid-users&m=107185287327641&w=2

To test the perl script, simply su to the same user as the proxy and execute perl /path/to/script/squid.pl. This will let you know if there are any issues with executing the script.

[jaromil]

did try that, script starts fine and "runs", basically doing nothing. Not sure how I can feed it urls..

will try squid -NDd1 now

[antagonismorg]

Any luck with running squid in debug?

[jaromil]

made some progresses in today's last commit.
(there were some dowse related trivial permission problems to fix)

scaffolding debug works by piping urls into the generater perl handler,
which takes some time to startup obviously, but then spits out a proper result.

lowering the children spawned it runs squid, but still when started from squid I cannot see results.
started from commandline works.

I believe today's commits have removed all dowse related problems.
are you running a working setup of this? which version of squid is used then?

[antagonismorg]

I am running, 2.7.STABLE9-4.1 for packages on Squeeze. I have the latest https-everywhere and perl-HTTPSEverywhere git repos.

[jaromil]

regarding this issue, I have a solution proposal that would benefit more than one software: can we make that perl script input the patterns into a key/value store like redis? if yes, the thing can really fly. it would fill it at start and then even more than an app can query it.

[jaromil]

hi P. would you be interested in revamping this discussion? I see EFF still regards https-everywhere as a priority. I'm quite sure it wouldn't take much to refactor the perl parser in order to have it input values into a fast k/v storage like Redis, something that would make the mechanism much faster. I'm willing to work on this for Dowse and it would be much more fun to do it together with you.

[antagonismorg]

I would be delighted to work with you on this. We might also want to include Mike Cardwell the original author of perl-https-everywhere as well. Drop me an email at marlowe@antagonism.org and we can get started.

[parazyd]

@jaromil Shall we close this?