A forward proxy with caching.
"I was struck by the thought that every word I spoke, every expression of my face or motion of my hand would endure in his implacable memory..."
Uses: https://github.com/chobits/ngx_http_proxy_connect_module/tree/96ae4e06381f821218f368ad0ba964f87cbe0266
(code copied to ngx_http_proxy_connect_module). Code is copied instead of using submodule as Github doesn't support including submodule code in releases.
faketimeopenssl(tested with version 1.0.2g)
funes listens on ports 80, 443, and 3128. By default it will cache all GET requests with a TTL of 5s, with certain content types having longer expirations (see conf/nginx.conf.server). It will serve stale responses indefinitely if there is no network connection.
make
This will compile Nginx and place all required files in the build directory.
Inside the build directory, bash run.sh to start the application.
There are several helper scripts that are placed in in the build directory. These can be run manually or added to crontabs.
logtruncate.sh- Truncates all logs from the Funes log directory.
clear_cert_disk_cache.sh <AGE_MIN, default 1440> <CACHE_DIR, default /data/funes/cert_cache>- Clears any generated SSL cert files older than
AGE_MINfromCACHE_DIR.
- Clears any generated SSL cert files older than
make package can be run to zip the build directory to package/funes.zip. The zip can be extracted and run elsewhere.
The following build options are available using environment variables:
# Disable the transparent proxy on port 80/443
DISABLE_TRANSPARENT_PROXY
# Disallow all connections except those from 127.0.0.1/24
RESTRICT_LOCAL
# Disallow all connections except from 127.0.0.1/24 and Docker IPs (172.18, 172.19, 172.21 prefixes)
# Cannot be combined with RESTRICT_LOCAL
RESTRICT_LOCAL_DOCKER
# Disable dynamic SSL cert generation, instead passing the root cert on each response
DISABLE_DYNAMIC_CERTSUsage example:
DISABLE_TRANSPARENT_PROXY=1 DISABLE_DYNAMIC_CERTS=1 make
DISABLE_TRANSPARENT_PROXY=1 RESTRICT_LOCAL=1 make
DISABLE_TRANSPARENT_PROXY=1 RESTRICT_LOCAL_DOCKER=1 makedocker build -f Dockerfile.build .
docker run -d -p 3128:3128 <created_image_id>
make dev-build # Builds the docker container
make dev # Starts container and runs Funes.
will start a local development instance in Docker.
If you have not made any changes between runs of the Docker container, you can run make dev again to avoid having to rebuild Openresty.
After making changes, run make dev-build again to rebuild the container and pull in changes.
For HTTPS requests, funes will generate certificates signed by its root CA certificate using OpenSSL. This is similar to other HTTPS proxies such as mitmproxy. The root CA cert+key used may be specified via ROOT_CA_CERT and ROOT_CA_KEY env vars when starting funes, or is autogenerated and printed to stdout. If the root CA cert is added to the client's local certificate store, certificate checking may be left on for the client.
The root CA cert+key used may be specified via ROOT_CA_CERT and ROOT_CA_KEY env vars when starting funes
If run with default settings, you will see the generated root CA cert in Docker stdout, e.g.:
-----BEGIN CERTIFICATE-----
MIIDzzCCAregAwIBAgIJAL0LMN2/s8xBMA0GCSqGSIb3DQEBCwUAMH4xCzAJBgNV
...
...
-----END CERTIFICATE-----
You may copy everything in this block into a file with a .crt extension and add it to your local system's certificates. For example, on Debian/Ubuntu you can run:
sudo cp <COPIED_CERT_FILE>.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates
Or on Mac: https://www.eduhk.hk/ocio/content/faq-how-add-root-certificate-mac-os-x
Run the full test suite:
make test
Due to Codeship limitations, it's not possible to run the full test suite in Codeship because you cannot disable the network adapter.
Full test suite should be run with RUN_CONTEXT=local after changes to test offline functionality.
In conf/nginx.conf.server, expiration rules can be set for URI ($uri_expiry), host ($host_expiry), and content type ($content_type_expiry). By default there are only content type expiry rules defined. Refer to the examples in this file for usage patterns.
Start chrome with the following flag:
--proxy-server="https=127.0.0.1:3128;http=127.0.0.1:3128"
Wrap your call to <electron_window>.loadURL following this example:
mainWindow.webContents.session.setProxy({proxyRules:"https=192.168.99.100:3128;http=192.168.99.100:3128"}, function () {
mainWindow.loadURL(url.format({
pathname: path.join(__dirname, 'index.html'),
protocol: 'file:',
slashes: true
}))
});
funes is meant to be used as a caching proxy for browser clients that need to function seamlessly if offline.
When Chrome is configured to use a proxy <proxy_host>:<proxy_port>, it does the following
- Sends HTTP requests to
<proxy_host>:<proxy_port>instead of<external_host>:80
- This request will contain a header
Host: <external_host
- Send a
CONNECTrequest to<proxy_host>:<proxy_port>, which opens a tunnel to the proxy. Send the original request through the tunnel.
funes responds to these requests:
- Send the request to
<external_host>:80. Cache the result and return. - Open a tunnel to
<funes_host>:443, which then receives the original HTTPS request. Send the request to<external_host>:443. Cache the result and return.
See https://en.wikipedia.org/wiki/HTTP_tunnel#HTTP_CONNECT_tunneling for more information on how tunneling works.
By default, the proxy server will cache everything with a TTL of 5s. If the resource cannot be refreshed after the TTL, it will continue serving the stale cached copy.
The main proxy server uses Nginx, and the Nginx Proxy Connect module to implement HTTP CONNECT method handling (since Nginx does not natively support the CONNECT method).
To redirect all HTTPS requests received by the tunnel back to Nginx, a local Although dnsmasq works well for this purpose, we now use the dnsmasq instance is started that responds to all DNS requests with 127.0.0.1. This local dnsmasq is configured as the DNS resolver for the proxy.proxy_connect_address directive provided by the Nginx Proxy Connect module, which redirects all proxied traffic to the specified address.
- Basic proxy setup taken from: https://stackoverflow.com/questions/46060028/how-to-use-nginx-as-forward-proxy-for-any-requested-location
- Cache config for proxy_cache: https://www.nginx.com/blog/nginx-caching-guide/
The cacert.pem file needs to be periodically updated to contain the latest public certificate authorities. If not you may eventually see SSL failures for some domains that result in the following error:
upstream SSL certificate verify error: (20:unable to get local issuer certificate)
This can happen if the cert of the upstream server was created with a CA that does not exist in cacert.pem.
To update cacert.pem run the following command:
perl scripts/mk-ca-bundle.plCopy the resulting ca-bundle.crt file to certs/cacert.pem.