Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

All branches: Bug with "space tab" in start-squid.sh #3

Closed
McBane87 opened this issue Feb 11, 2021 · 8 comments
Closed

All branches: Bug with "space tab" in start-squid.sh #3

McBane87 opened this issue Feb 11, 2021 · 8 comments

Comments

@McBane87
Copy link
Owner

my squid.conf

http_port 0.0.0.0:46001
icp_port 0

acl localnet src 0.0.0.1-0.255.255.255	        # RFC 1122 "this" network (LAN)
acl localnet src 10.0.0.0/8			# RFC 1918 local private network (LAN)
acl localnet src 100.64.0.0/10			# RFC 6598 shared address space (CGN)
acl localnet src 169.254.0.0/16 		# RFC 3927 link-local (directly plugged) machines
acl localnet src 172.16.0.0/12			# RFC 1918 local private network (LAN)
acl localnet src 192.168.0.0/16			# RFC 1918 local private network (LAN)
acl localnet src fc00::/7       		# RFC 4193 local private network range
acl localnet src fe80::/10      		# RFC 4291 link-local (directly plugged) machines

acl SSL_ports port 443
acl Safe_ports port 80			# http
acl Safe_ports port 21			# ftp
acl Safe_ports port 22			# ssh
acl Safe_ports port 443			# https
acl CONNECT method CONNECT

auth_param basic program /usr/lib/squid/basic_ncsa_auth /etc/squid/.htpasswd
auth_param basic children 15
auth_param basic realm Welcome to Proxy Server
auth_param basic credentialsttl 5 hours
auth_param basic casesensitive on
acl auth proxy_auth REQUIRED
http_access allow auth

http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports

http_access allow localhost manager
http_access deny manager

http_access allow localnet
http_access allow localhost

http_access deny all

cache_dir ufs /var/spool/squid 500 16 256
coredump_dir /var/spool/squid

cache_effective_user proxy

refresh_pattern ^ftp:		1440	20%	10080
refresh_pattern ^gopher:	1440	0%	1440
refresh_pattern -i (/cgi-bin/|\?) 0	0%	0
refresh_pattern .		0	20%	4320

negative_ttl 0 
dns_v4_first on
#dns_nameservers 1.1.1.1
#dns_nameservers 8.8.8.8 8.8.4.4

positive_dns_ttl 8 hours
negative_dns_ttl 30 seconds

pinger_enable off
half_closed_clients off
quick_abort_min 0 KB
quick_abort_max 0 KB
quick_abort_pct 95

client_persistent_connections off
server_persistent_connections off

visible_hostname unknown
via off
forwarded_for transparent
httpd_suppress_version_string on

request_header_access Allow allow all 
request_header_access Authorization allow all 
request_header_access WWW-Authenticate allow all 
request_header_access Proxy-Authorization allow all 
request_header_access Proxy-Authenticate allow all 
request_header_access Cache-Control allow all 
request_header_access Content-Encoding allow all 
request_header_access Content-Length allow all 
request_header_access Content-Type allow all 
request_header_access Date allow all 
request_header_access Expires allow all 
request_header_access Host allow all 
request_header_access If-Modified-Since allow all 
request_header_access Last-Modified allow all 
request_header_access Location allow all 
request_header_access Pragma allow all 
request_header_access Accept allow all 
request_header_access Accept-Charset allow all 
request_header_access Accept-Encoding allow all 
request_header_access Accept-Language allow all 
request_header_access Content-Language allow all 
request_header_access Mime-Version allow all 
request_header_access Retry-After allow all 
request_header_access Title allow all 
request_header_access Connection allow all 
request_header_access Proxy-Connection allow all 
request_header_access User-Agent allow all 
request_header_access Cookie allow all 
request_header_access All deny all

my docker-compose.yml

version: '3.3'
services:
    squid:
        ports:
            - '46001:46001'
        image: 'distahl/squid:latest'
        volumes:
            - './config:/etc/squid'
            - './../squid_log:/var/log/squid' 
            - './../squid_cache:/var/spool/squid'
        environment:
            - PROXY_UID=1000
            - PROXY_GID=1000                  
        container_name: dsquid

1- host has user "squid" with UID=1000 & GID=1000
2- host volume directories owner and group set to "squid:squid" with UID=1000 & GID=1000
3- docker-compose up command executed under "squid" user with sudo privileges

sudo docker-compose up

the result

2021/02/11 16:02:03| Created PID file (/run/squid.pid)
2021/02/11 16:02:03| Set Current Directory to /var/spool/squid
2021/02/11 16:02:03| Starting Squid Cache version 4.13 for x86_64-pc-linux-gnu...
2021/02/11 16:02:03| Service Name: squid
2021/02/11 16:02:03| Process ID 37
2021/02/11 16:02:03| Process Roles: master worker
2021/02/11 16:02:03| With 65535 file descriptors available
2021/02/11 16:02:03| Initializing IP Cache...
2021/02/11 16:02:03| DNS Socket created at 0.0.0.0, FD 3
2021/02/11 16:02:03| Adding nameserver 127.0.0.11 from /etc/resolv.conf
2021/02/11 16:02:03| Adding ndots 1 from /etc/resolv.conf
2021/02/11 16:02:03| helperOpenServers: Starting 0/15 'basic_ncsa_auth' processes
2021/02/11 16:02:03| helperOpenServers: No 'basic_ncsa_auth' processes needed.
2021/02/11 16:02:03| Logfile: opening log daemon:/var/log/squid/access.log
2021/02/11 16:02:03| Logfile Daemon: opening log /var/log/squid/access.log
2021/02/11 16:02:04| Unlinkd pipe opened on FD 14
2021/02/11 16:02:04| Local cache digest enabled; rebuild/rewrite every 3600/3600 sec
2021/02/11 16:02:04| Store logging disabled
2021/02/11 16:02:04| Swap maxSize 512000 + 262144 KB, estimated 59549 objects
2021/02/11 16:02:04| Target number of buckets: 2977
2021/02/11 16:02:04| Using 8192 Store buckets
2021/02/11 16:02:04| Max Mem  size: 262144 KB
2021/02/11 16:02:04| Max Swap size: 512000 KB
2021/02/11 16:02:04| ERROR: /var/spool/squid/00: (2) No such file or directory
2021/02/11 16:02:04| Not currently OK to rewrite swap log.
2021/02/11 16:02:04| storeDirWriteCleanLogs: Operation aborted.
2021/02/11 16:02:04| FATAL: 	Failed to verify one of the swap directories, Check cache.log
	for details.  Run 'squid -z' to create swap directories
	if needed, or if running Squid for the first time.
2021/02/11 16:02:04| Squid Cache (Version 4.13): Terminated abnormally.
CPU Usage: 0.031 seconds = 0.014 user + 0.017 sys
Maximum Resident Size: 123808 KB
Page faults with physical i/o: 0
2021/02/11 16:02:04| Removing PID file (/run/squid.pid)

I even tried to run squid container with changed entrypoint to /bin/bash then run ./start-squid.sh with the same result
I think there is problem with the start-squid.sh when trying to create spool dirs because when I disable caching in squid.conf squid container run without problem and squid logs access.log && cache.log copied to host volume directory ./../squid_log

update:

I started the container with these lines commented in squid.conf to be able to access the conatiner bash

# cache_dir ufs /var/spool/squid 500 16 256
# coredump_dir /var/spool/squid
# cache_effective_user proxy

then after the container started I uncommented the lines in squid.conf on host volume directory ./config/squid.conf to enable the cache again then what I did

cd /var/spool/squid
/usr/sbin/squid -z -f /etc/squid/squid.conf

1- the spool directories created and copied to host volume directory ./../squid_cache
2- the owner of volume host directories is squid:squid
3- the owner of container directories proxy:proxy

which means start-squid.sh failed to create spool directories for some reason

I stoped squid container then

sudo docker-compose up

the result

2021/02/11 17:45:09| Created PID file (/run/squid.pid)
2021/02/11 17:45:09| Set Current Directory to /var/spool/squid
2021/02/11 17:45:09| Starting Squid Cache version 4.13 for x86_64-pc-linux-gnu...
2021/02/11 17:45:09| Service Name: squid
2021/02/11 17:45:09| Process ID 37
2021/02/11 17:45:09| Process Roles: master worker
2021/02/11 17:45:09| With 65535 file descriptors available
2021/02/11 17:45:09| Initializing IP Cache...
2021/02/11 17:45:09| DNS Socket created at 0.0.0.0, FD 3
2021/02/11 17:45:09| Adding nameserver 127.0.0.11 from /etc/resolv.conf
2021/02/11 17:45:09| Adding ndots 1 from /etc/resolv.conf
2021/02/11 17:45:09| helperOpenServers: Starting 0/15 'basic_ncsa_auth' processes
2021/02/11 17:45:09| helperOpenServers: No 'basic_ncsa_auth' processes needed.
2021/02/11 17:45:09| Logfile: opening log daemon:/var/log/squid/access.log
2021/02/11 17:45:09| Logfile Daemon: opening log /var/log/squid/access.log
2021/02/11 17:45:09| Unlinkd pipe opened on FD 14
2021/02/11 17:45:09| Local cache digest enabled; rebuild/rewrite every 3600/3600 sec
2021/02/11 17:45:09| Store logging disabled
2021/02/11 17:45:09| Swap maxSize 512000 + 262144 KB, estimated 59549 objects
2021/02/11 17:45:09| Target number of buckets: 2977
2021/02/11 17:45:09| Using 8192 Store buckets
2021/02/11 17:45:09| Max Mem  size: 262144 KB
2021/02/11 17:45:09| Max Swap size: 512000 KB
2021/02/11 17:45:09| Rebuilding storage in /var/spool/squid (no log)
2021/02/11 17:45:09| Using Least Load store dir selection
2021/02/11 17:45:09| Set Current Directory to /var/spool/squid
2021/02/11 17:45:09| Finished loading MIME types and icons.
2021/02/11 17:45:09| HTCP Disabled.
2021/02/11 17:45:09| Squid plugin modules loaded: 0
2021/02/11 17:45:09| Adaptation support is off.
2021/02/11 17:45:09| Accepting HTTP Socket connections at local=0.0.0.0:46001 remote=[::] FD 16 flags=9
2021/02/11 17:45:09| Configuring Parent dgost1/49152/0
2021/02/11 17:45:09| Configuring Parent dgost2/49153/0
2021/02/11 17:45:09| Configuring Parent dgost3/49154/0
2021/02/11 17:45:09| Configuring Parent dgost4/49155/0
2021/02/11 17:45:09| Configuring Parent dgost5/49156/0
2021/02/11 17:45:09| Configuring Parent dgost6/49157/0
2021/02/11 17:45:09| Done scanning /var/spool/squid dir (0 entries)
2021/02/11 17:45:09| Finished rebuilding storage from disk.
2021/02/11 17:45:09|         0 Entries scanned
2021/02/11 17:45:09|         0 Invalid entries.
2021/02/11 17:45:09|         0 With invalid flags.
2021/02/11 17:45:09|         0 Objects loaded.
2021/02/11 17:45:09|         0 Objects expired.
2021/02/11 17:45:09|         0 Objects cancelled.
2021/02/11 17:45:09|         0 Duplicate URLs purged.
2021/02/11 17:45:09|         0 Swapfile clashes avoided.
2021/02/11 17:45:09|   Took 0.14 seconds (  0.00 objects/sec).
2021/02/11 17:45:09| Beginning Validation Procedure
2021/02/11 17:45:09|   Completed Validation Procedure
2021/02/11 17:45:09|   Validated 0 Entries
2021/02/11 17:45:09|   store_swap_size = 0.00 KB
2021/02/11 17:45:10| storeLateRelease: released 0 objects

Originally posted by @srcKod in #2 (comment)
@McBane87
Copy link
Owner Author

Thanks for reporting. Looks like the start script is outdated. I have to copy newer versions of find_cache_dir () and grepconf () from /etc/init.d/squid . After that it should work.

@McBane87 McBane87 changed the title start-squid.sh outdated in master & bullseye branch All branches: Bug with "space tab" in start-squid.sh Feb 11, 2021
@McBane87
Copy link
Owner Author

I have to correct myself. The script isn't outdated. I just somehow screwed up the "space tab" used in both functions.

@McBane87
Copy link
Owner Author

McBane87 commented Feb 11, 2021 •
edited
Loading

Should be fixed with aee7117
Docker rebuild will be triggered now. But it can take some time until it will be updated.

@srcKod
Copy link

srcKod commented Feb 11, 2021

last image downloaded. the result when new container created

2021/02/11 22:54:31| Created PID file (/run/squid.pid)
2021/02/11 22:54:31| FATAL: Squid is already running: Found fresh instance PID file (/run/squid.pid) with PID 56
    exception location: Instance.cc(121) ThrowIfAlreadyRunningWith

@McBane87
Copy link
Owner Author

McBane87 commented Feb 12, 2021 •
edited
Loading

Sorry, I think now it should work. Fix is 789a5d3
Rebuild of docker images is started.

Additionally:
You most likely have to clear your cache directory before starting again. Otherwise the initial creation of the files will fail.

@srcKod
Copy link

srcKod commented Feb 12, 2021 •
edited
Loading

I downloaded bullseye image. now it can build the cache directories
note that cache storage scheme must be ufs and it will produce errors if the cache storage scheme set to aufs in squid.conf

here the cache storage scheme set to aufs

2021/02/12 08:46:24| Created PID file (/run/squid.pid)
2021/02/12 08:46:24| Created PID file (/run/squid.pid)
2021/02/12 08:46:24 kid1| Set Current Directory to /var/spool/squid
2021/02/12 08:46:24 kid1| Creating missing swap directories
2021/02/12 08:46:24| Set Current Directory to /var/spool/squid
2021/02/12 08:46:24 kid1| /var/spool/squid exists
2021/02/12 08:46:24 kid1| Making directories in /var/spool/squid/00
2021/02/12 08:46:24| Starting Squid Cache version 4.13 for x86_64-pc-linux-gnu...
2021/02/12 08:46:24| Service Name: squid
2021/02/12 08:46:24| Process ID 61
2021/02/12 08:46:24| Process Roles: master worker
2021/02/12 08:46:24| With 65535 file descriptors available
2021/02/12 08:46:24| Initializing IP Cache...
2021/02/12 08:46:24| DNS Socket created at 0.0.0.0, FD 3
2021/02/12 08:46:24| Adding nameserver 127.0.0.11 from /etc/resolv.conf
2021/02/12 08:46:24| Adding ndots 1 from /etc/resolv.conf
2021/02/12 08:46:24| helperOpenServers: Starting 0/15 'basic_ncsa_auth' processes
2021/02/12 08:46:24| helperOpenServers: No 'basic_ncsa_auth' processes needed.
2021/02/12 08:46:24 kid1| Making directories in /var/spool/squid/01
2021/02/12 08:46:24| Logfile: opening log daemon:/var/log/squid/access.log
2021/02/12 08:46:24| Logfile Daemon: opening log /var/log/squid/access.log
2021/02/12 08:46:24 kid1| Making directories in /var/spool/squid/02
2021/02/12 08:46:24| Local cache digest enabled; rebuild/rewrite every 3600/3600 sec
2021/02/12 08:46:24| Store logging disabled
2021/02/12 08:46:24| Swap maxSize 512000 + 262144 KB, estimated 59549 objects
2021/02/12 08:46:24| Target number of buckets: 2977
2021/02/12 08:46:24| Using 8192 Store buckets
2021/02/12 08:46:24| Max Mem  size: 262144 KB
2021/02/12 08:46:24| Max Swap size: 512000 KB
2021/02/12 08:46:24| ERROR: /var/spool/squid/03: (2) No such file or directory
2021/02/12 08:46:24| Not currently OK to rewrite swap log.
2021/02/12 08:46:24| storeDirWriteCleanLogs: Operation aborted.
2021/02/12 08:46:24| FATAL: 	Failed to verify one of the swap directories, Check cache.log
	for details.  Run 'squid -z' to create swap directories
	if needed, or if running Squid for the first time.
2021/02/12 08:46:24| Squid Cache (Version 4.13): Terminated abnormally.
CPU Usage: 0.035 seconds = 0.018 user + 0.018 sys
Maximum Resident Size: 122864 KB
Page faults with physical i/o: 2
2021/02/12 08:46:24| Removing PID file (/run/squid.pid)
2021/02/12 08:46:24 kid1| Making directories in /var/spool/squid/03

@McBane87
Copy link
Owner Author

Aaaaand another bugfix -.-
206ed4b

@srcKod
Copy link

srcKod commented Feb 12, 2021

the last build of bullseye image now success with aufs

2021/02/12 10:58:12| Created PID file (/run/squid.pid)
2021/02/12 10:58:12 kid1| Set Current Directory to /var/spool/squid
2021/02/12 10:58:12 kid1| Creating missing swap directories
2021/02/12 10:58:12 kid1| /var/spool/squid exists
2021/02/12 10:58:12 kid1| Making directories in /var/spool/squid/00
2021/02/12 10:58:12 kid1| Making directories in /var/spool/squid/01
2021/02/12 10:58:12 kid1| Making directories in /var/spool/squid/02
2021/02/12 10:58:12 kid1| Making directories in /var/spool/squid/03
2021/02/12 10:58:12 kid1| Making directories in /var/spool/squid/04
2021/02/12 10:58:12 kid1| Making directories in /var/spool/squid/05
2021/02/12 10:58:12 kid1| Making directories in /var/spool/squid/06
2021/02/12 10:58:12 kid1| Making directories in /var/spool/squid/07
2021/02/12 10:58:12 kid1| Making directories in /var/spool/squid/08
2021/02/12 10:58:12 kid1| Making directories in /var/spool/squid/09
2021/02/12 10:58:12 kid1| Making directories in /var/spool/squid/0A
2021/02/12 10:58:12 kid1| Making directories in /var/spool/squid/0B
2021/02/12 10:58:12 kid1| Making directories in /var/spool/squid/0C
2021/02/12 10:58:12 kid1| Making directories in /var/spool/squid/0D
2021/02/12 10:58:12 kid1| Making directories in /var/spool/squid/0E
2021/02/12 10:58:12 kid1| Making directories in /var/spool/squid/0F
2021/02/12 10:58:12| Removing PID file (/run/squid.pid)
2021/02/12 10:58:12| Created PID file (/run/squid.pid)
2021/02/12 10:58:12| Set Current Directory to /var/spool/squid
2021/02/12 10:58:12| Starting Squid Cache version 4.13 for x86_64-pc-linux-gnu...
2021/02/12 10:58:12| Service Name: squid
2021/02/12 10:58:12| Process ID 62
2021/02/12 10:58:12| Process Roles: master worker
2021/02/12 10:58:12| With 65535 file descriptors available
2021/02/12 10:58:12| Initializing IP Cache...
2021/02/12 10:58:12| DNS Socket created at 0.0.0.0, FD 3
2021/02/12 10:58:12| Adding nameserver 127.0.0.11 from /etc/resolv.conf
2021/02/12 10:58:12| Adding ndots 1 from /etc/resolv.conf
2021/02/12 10:58:12| helperOpenServers: Starting 0/15 'basic_ncsa_auth' processes
2021/02/12 10:58:12| helperOpenServers: No 'basic_ncsa_auth' processes needed.
2021/02/12 10:58:12| Logfile: opening log daemon:/var/log/squid/access.log
2021/02/12 10:58:12| Logfile Daemon: opening log /var/log/squid/access.log
2021/02/12 10:58:12| Local cache digest enabled; rebuild/rewrite every 3600/3600 sec
2021/02/12 10:58:12| Store logging disabled
2021/02/12 10:58:12| Swap maxSize 512000 + 25600 KB, estimated 41353 objects
2021/02/12 10:58:12| Target number of buckets: 2067
2021/02/12 10:58:12| Using 8192 Store buckets
2021/02/12 10:58:12| Max Mem  size: 25600 KB
2021/02/12 10:58:12| Max Swap size: 512000 KB
2021/02/12 10:58:12| Rebuilding storage in /var/spool/squid (no log)
2021/02/12 10:58:12| Using Least Load store dir selection
2021/02/12 10:58:12| Set Current Directory to /var/spool/squid
2021/02/12 10:58:12| Finished loading MIME types and icons.
2021/02/12 10:58:12| HTCP Disabled.
2021/02/12 10:58:12| Squid plugin modules loaded: 0
2021/02/12 10:58:12| Adaptation support is off.
2021/02/12 10:58:12| Accepting HTTP Socket connections at local=0.0.0.0:46001 remote=[::] FD 12 flags=9
2021/02/12 10:58:13| Configuring Parent dgost1/49152/0
2021/02/12 10:58:13| Configuring Parent dgost6/49157/0
2021/02/12 10:58:13| Configuring Parent dgost2/49153/0
2021/02/12 10:58:13| Configuring Parent dgost3/49154/0
2021/02/12 10:58:13| Configuring Parent dgost4/49155/0
2021/02/12 10:58:13| Configuring Parent dgost5/49156/0
2021/02/12 10:58:13| Done scanning /var/spool/squid dir (0 entries)
2021/02/12 10:58:13| Finished rebuilding storage from disk.
2021/02/12 10:58:13|         0 Entries scanned
2021/02/12 10:58:13|         0 Invalid entries.
2021/02/12 10:58:13|         0 With invalid flags.
2021/02/12 10:58:13|         0 Objects loaded.
2021/02/12 10:58:13|         0 Objects expired.
2021/02/12 10:58:13|         0 Objects cancelled.
2021/02/12 10:58:13|         0 Duplicate URLs purged.
2021/02/12 10:58:13|         0 Swapfile clashes avoided.
2021/02/12 10:58:13|   Took 0.11 seconds (  0.00 objects/sec).
2021/02/12 10:58:13| Beginning Validation Procedure
2021/02/12 10:58:13|   Completed Validation Procedure
2021/02/12 10:58:13|   Validated 0 Entries
2021/02/12 10:58:13|   store_swap_size = 0.00 KB
2021/02/12 10:58:13| Starting new basicauthenticator helpers...
2021/02/12 10:58:13| helperOpenServers: Starting 1/15 'basic_ncsa_auth' processes
2021/02/12 10:58:13| storeLateRelease: released 0 objects

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@McBane87 @srcKod