Skip to content

Commit

Permalink
Updating docs and dockerfiles
Browse files Browse the repository at this point in the history
  • Loading branch information
@yifeikong
yifeikong committed Apr 17, 2024
1 parent efd4a72 commit 67828d8
Show file tree
Hide file tree
Showing 9 changed files with 39 additions and 225 deletions.
91 changes: 15 additions & 76 deletions INSTALL.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ There are currently three build options depending on your use case:
* [Cross compiling](#cross-compiling) using an autotools-based Makefile
* [Docker container build](#docker-build)

There are two versions of `curl-impersonate` for technical reasons. The **chrome** version is used to impersonate Chrome, Edge and Safari. The **firefox** version is used to impersonate Firefox.
Unlike the upstream project, there is only one version in this fork, namely the Chrome version, for impersonating all main stream browsers.

## Native build

Expand All @@ -18,18 +18,14 @@ Install dependencies for building all the components:

```sh
sudo apt install build-essential pkg-config cmake ninja-build curl autoconf automake libtool
# For the Firefox version only
sudo apt install python3-pip libnss3
pip install gyp-next
export PATH="$PATH:~/.local/bin" # Add gyp to PATH
# For the Chrome version only
sudo apt install golang-go unzip
sudo apt install zstd libzstd-dev
```

Clone this repository:

```sh
git clone https://github.com/lwthiker/curl-impersonate.git
git clone https://github.com/yifeikong/curl-impersonate.git
cd curl-impersonate
```

Expand All @@ -38,10 +34,7 @@ Configure and compile:
```sh
mkdir build && cd build
../configure
# Build and install the Firefox version
make firefox-build
sudo make firefox-install
# Build and install the Chrome version
# Build and install
make chrome-build
sudo make chrome-install
# You may need to update the linker's cache to find libcurl-impersonate
Expand All @@ -55,14 +48,12 @@ This will install curl-impersonate, libcurl-impersonate and the wrapper scripts
After installation you can run the wrapper scripts, e.g.:

```sh
curl_ff98 https://www.wikipedia.org
curl_chrome99 https://www.wikipedia.org
curl_chrome119 https://www.wikipedia.org
```

or run directly with you own flags:

```sh
curl-impersonate-ff https://www.wikipedia.org
curl-impersonate-chrome https://www.wikipedia.org
```

Expand All @@ -78,13 +69,7 @@ yum install cmake3 python3 python3-pip
yum install ninja-build
# OR
pip3 install ninja
```

For the Firefox version, install NSS and gyp:

```sh
yum install nss nss-pem
pip3 install gyp-next
yum install zstd libzstd-devel
```

For the Chrome version, install Go.
Expand All @@ -102,16 +87,14 @@ Install dependencies for building all the components:

```sh
brew install pkg-config make cmake ninja autoconf automake libtool
# For the Firefox version only
brew install sqlite nss
pip3 install gyp-next
# For the Chrome version only
brwe install zstd
brew install go
```

Clone this repository:
```
git clone https://github.com/lwthiker/curl-impersonate.git

```sh
git clone https://github.com/yifeikong/curl-impersonate.git
cd curl-impersonate
```

Expand All @@ -120,10 +103,7 @@ Configure and compile:
```sh
mkdir build && cd build
../configure
# Build and install the Firefox version
gmake firefox-build
sudo gmake firefox-install
# Build and install the Chrome version
# Build and install
gmake chrome-build
sudo gmake chrome-install
# Optionally remove all the build files
Expand All @@ -134,33 +114,14 @@ cd ../ && rm -Rf build

To compile curl-impersonate statically with libcurl-impersonate, pass `--enable-static` to the `configure` script.

### A note about the Firefox version

The Firefox version compiles a static version of nss, Firefox's TLS library.
For NSS to have a list of root certificates, curl attempts to load at runtime `libnssckbi`, one of the NSS libraries.
If you get the error:

```sh
curl: (60) Peer's Certificate issuer is not recognized
```
or
```sh
curl: (77) Problem with the SSL CA cert (path? access rights?)
```
, make sure that NSS is installed (see above).
If the issue persists it might be that NSS is installed in a non-standard location on your system.
Please open an issue in that case.

## Cross compiling

There is some basic support for cross compiling curl-impersonate.
It is currently being used to build curl-impersonate for ARM64 (aarch64) systems from x86-64 systems.
Cross compiling is similar to the usual build but a bit trickier:

* You'd have to build zlib for the target architecture so that curl can link with it.
* You'd have to build zlib and zstd for the target architecture so that curl can link with it.
* Some paths have to be specified manually since curl's own build system can't determine their location.

An example build for aarch64 on Ubuntu x86_64:
Expand All @@ -170,25 +131,21 @@ sudo apt-get install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu

./configure --host=aarch64-linux-gnu \
--with-zlib=/path/to/compiled/zlib \
--with-zstd=/path/to/compiled/zstd \
--with-ca-path=/etc/ssl/certs \
--with-ca-bundle=/etc/ssl/certs/ca-certificates.crt \
--with-libnssckbi=/usr/lib/aarch64-linux-gnu/nss
--with-ca-bundle=/etc/ssl/certs/ca-certificates.crt

make chrome-build
make firefox-build
```

The flags mean as follows:
`--with-zlib` is the location of a compiled zlib library for the target architecture.
`--with-zlib/zstd` is the location of a compiled zlib/zstd library for the target architecture.
`--with-ca-path` and `--with-ca-bundle` will be passed to curl's configure script as is.
`--with-libnssckbi` indicates the location of libnssckbi.so on the target system. This file contains the certificates needed by curl. This must be supplied if NSS is not installed in a standard location (i.e. not in `/usr/lib`).

## Docker build

The Docker build is a bit more reproducible and serves as the reference implementation. It creates a Debian-based Docker image with the binaries.

### Chrome version
[`chrome/Dockerfile`](chrome/Dockerfile) is a debian-based Dockerfile that will build curl with all the necessary modifications and patches. Build it like the following:

```sh
Expand All @@ -202,21 +159,3 @@ The resulting binaries and libraries are in the `/usr/local` directory, which co
* `libcurl-impersonate-chrome.so`, `libcurl-impersonate.so` - libcurl compiled with impersonation support. See [libcurl-impersonate](README.md#libcurl-impersonate) for more details.

You can use them inside the docker, copy them out using `docker cp` or use them in a multi-stage docker build.

### Firefox version

Build with:

```sh
docker build -t curl-impersonate-ff firefox/
```

The resulting binaries and libraries are in the `/usr/local` directory, which contains:

* `curl-impersonate-ff`, `curl-impersonate` - The curl binary that can impersonate Firefox. It is compiled statically against libcurl, nss, and libnghttp2 so that it won't conflict with any existing libraries on your system. You can use it from the container or copy it out. Tested to work on Ubuntu 20.04.
* `curl_ff91esr`, `curl_ff95`, `...` - Wrapper scripts that launch `curl-impersonate` with all the needed flags.
* `libcurl-impersonate-ff.so`, `libcurl-impersonate.so` - libcurl compiled with impersonation support. See [libcurl-impersonate](README.md#libcurl-impersonate) for more details.
If you use it outside the container, install the following dependency:
* `sudo apt install libnss3`. Even though nss is statically compiled into `curl-impersonate`, it is still necessary to install libnss3 because curl dynamically loads `libnssckbi.so`, a file containing Mozilla's list of trusted root certificates. Alternatively, use `curl -k` to disable certificate verification.
1 change: 1 addition & 0 deletions LICENSE
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
MIT License

Copyright (c) 2022 lwthiker
Copyright (c) 2024 yifeikong

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
Expand Down
111 changes: 0 additions & 111 deletions Makefile.in
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,6 @@ SUBJOBS := 4

BROTLI_VERSION := 1.0.9
# In case this is changed, update build-and-test-make.yml as well
NSS_VERSION := nss-3.92
NSS_URL := https://ftp.mozilla.org/pub/security/nss/releases/NSS_3_92_RTM/src/nss-3.92-with-nspr-4.35.tar.gz
# In case this is changed, update build-and-test-make.yml as well
BORING_SSL_COMMIT := d24a38200fef19150eef00cad35b138936c08767
NGHTTP2_VERSION := nghttp2-1.56.0
Expand All @@ -21,15 +19,11 @@ CURL_VERSION := curl-8.5.0

brotli_install_dir := $(abspath brotli-$(BROTLI_VERSION)/out/installed)
brotli_static_libs := $(brotli_install_dir)/lib/libbrotlicommon-static.a $(brotli_install_dir)/lib/libbrotlidec-static.a
nss_install_dir := $(abspath $(NSS_VERSION)/dist/Release)
nss_static_libs := $(nss_install_dir)/lib/libnss_static.a
boringssl_install_dir := $(abspath boringssl/build)
boringssl_static_libs := $(boringssl_install_dir)/lib/libssl.a $(boringssl_install_dir)/lib/libcrypto.a
nghttp2_install_dir := $(abspath $(NGHTTP2_VERSION)/installed)
nghttp2_static_libs := $(nghttp2_install_dir)/lib/libnghttp2.a

# Dependencies needed to compile the Firefox version
firefox_libs := $(brotli_static_libs) $(nss_static_libs) $(nghttp2_static_libs)
# Dependencies needed to compile the Chrome version
chrome_libs := $(brotli_static_libs) $(boringssl_static_libs) $(nghttp2_static_libs)

Expand Down Expand Up @@ -115,7 +109,6 @@ chrome-clean: ## Clean build artifacts of the Chrome version. Use after re-runni

clean: ## Remove all build artifacts, including dependencies
rm -Rf brotli-$(BROTLI_VERSION).tar.gz brotli-$(BROTLI_VERSION)
rm -Rf $(NSS_VERSION).tar.gz $(NSS_VERSION)
rm -Rf boringssl.zip boringssl
rm -Rf $(NGHTTP2_VERSION).tar.bz2 $(NGHTTP2_VERSION)
rm -Rf $(CURL_VERSION).tar.xz $(CURL_VERSION)
Expand Down Expand Up @@ -156,54 +149,6 @@ $(brotli_static_libs): brotli-$(BROTLI_VERSION).tar.gz
@cmake@ --build . --config Release --target install --parallel $(SUBJOBS)


$(NSS_VERSION).tar.gz:
curl -L -o $(NSS_VERSION).tar.gz $(NSS_URL)

$(nss_static_libs): $(NSS_VERSION).tar.gz
tar xf $(NSS_VERSION).tar.gz

ifeq ($(host),$(build))
# Native build, use NSS' build script.
cd $(NSS_VERSION)/nss
./build.sh -o --disable-tests --static --python=python3 -j $(SUBJOBS)
else
# We are cross compiling.
# Cross compiling NSS is not supported by its build script and is poorly
# documented. We need to compile NSPR manually and only then compile nss.
case $(host_cpu) in \
*64*) \
use_64="1"; \
nspr_configure_flags="--enable-64bit"; \
;; \
*) \
use_64="0"; \
nspr_configure_flags=""; \
;; \
esac

# Cross-compile nspr separately
cd $(NSS_VERSION)/nspr
./configure --prefix=$(nss_install_dir) \
--disable-debug --enable-optimize \
--target=$(host_alias) \
$$nspr_configure_flags
$(MAKE) MAKEFLAGS=
$(MAKE) install MAKEFLAGS=

# Now we can run ./build.sh with the already built nspr
cd ../nss
CC=$(CC) CXX=$(CXX) CCC=$(CXX) \
./build.sh -o --disable-tests --static --python=python3 \
--with-nspr=$(nss_install_dir)/include/nspr:$(nss_install_dir)/lib \
--target=$(host_cpu) \
-Duse_system_zlib=0 \
-Dsign_libs=0
endif
# Hack for macOS: Remove dynamic libraries to force the linker to use the
# static ones when linking curl.
rm -Rf $(nss_install_dir)/lib/*.dylib


boringssl.zip:
curl -L https://github.com/google/boringssl/archive/$(BORING_SSL_COMMIT).zip \
-o boringssl.zip
Expand Down Expand Up @@ -282,17 +227,6 @@ $(CURL_VERSION).tar.xz:
curl -L "https://curl.se/download/$(CURL_VERSION).tar.xz" \
-o "$(CURL_VERSION).tar.xz"

# Apply the "Firefox version" patches and mark using a dummy file
$(CURL_VERSION)/.patched-ff: $(srcdir)/firefox/patches/curl-*.patch
rm -Rf $(CURL_VERSION)
tar -xf $(CURL_VERSION).tar.xz
cd $(CURL_VERSION)
for p in $^; do patch -p1 < $$p; done
# Re-generate the configure script
autoreconf -fi
touch .patched-ff
rm -f .patched-chrome

# Apply the "Chorme version" patches and mark using a dummy file
$(CURL_VERSION)/.patched-chrome: $(srcdir)/chrome/patches/curl-*.patch
rm -Rf $(CURL_VERSION)
Expand All @@ -302,49 +236,6 @@ $(CURL_VERSION)/.patched-chrome: $(srcdir)/chrome/patches/curl-*.patch
# Re-generate the configure script
autoreconf -fi
touch .patched-chrome
rm -f .patched-ff

# This is a small hack that flags that curl was patched and configured in the "firefox" version
$(CURL_VERSION)/.firefox: $(firefox_libs) $(CURL_VERSION).tar.xz $(CURL_VERSION)/.patched-ff
cd $(CURL_VERSION)

# Set up the configure flags to curl.
# If the user provided the --host flag to our configure script
# (for cross compilation), then pass it on to curl.
{ \
config_flags="--prefix=@prefix@"; \
config_flags+=" --with-nghttp2=$(nghttp2_install_dir)"; \
config_flags+=" --with-brotli=$(brotli_install_dir)"; \
config_flags+=" --with-nss=$(nss_install_dir) --with-nss-deprecated"; \
config_flags+=" --enable-websockets"; \
config_flags+=" --enable-ipv6"; \
config_flags+=" USE_CURL_SSLKEYLOGFILE=true"; \
if test "$(static_build)" = "yes"; then \
config_flags+=" --enable-static --disable-shared"; \
fi; \
if test -n "$(host_alias)"; then \
config_flags+=" --host=$(host_alias)"; \
fi; \
if test -n "$(with_zlib)"; then \
config_flags+=" --with-zlib=$(with_zlib)"; \
else \
config_flags+=" --with-zlib"; \
fi; \
if test -n "$(with_libnssckbi)"; then \
config_flags+=" --with-libnssckbi=$(with_libnssckbi)"; \
fi; \
add_cflags="-I$(nss_install_dir)/../public/nss"; \
add_cflags+=" -I$(nss_install_dir)/include/nspr"; \
}

echo "Configuring curl with: $$config_flags"

./configure $$config_flags CFLAGS="$(CFLAGS) $$add_cflags"
# Remove possible leftovers from a previous compilation
$(MAKE) clean MAKEFLAGS=
touch .firefox
# Remove the Chrome flag if it exists
rm -f .chrome

# This is a small hack that flags that curl was patched and configured in the "chrome" version
$(CURL_VERSION)/.chrome: $(chrome_libs) $(CURL_VERSION).tar.xz $(CURL_VERSION)/.patched-chrome
Expand Down Expand Up @@ -394,5 +285,3 @@ $(CURL_VERSION)/.chrome: $(chrome_libs) $(CURL_VERSION).tar.xz $(CURL_VERSION)/.
# Remove possible leftovers from a previous compilation
$(MAKE) clean MAKEFLAGS=
touch .chrome
# Remove the Firefox flag if it exists
rm -f .firefox
2 changes: 1 addition & 1 deletion README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -195,12 +195,12 @@ In particular, see the [note about the Firefox version](INSTALL.md#a-note-about-
This repository contains these folders:
* [chrome](chrome) - Scripts and patches for building the Chrome version of `curl-impersonate`.
* [Dockerfile](chrome/Dockerfile) - Used to build `curl-impersonate` with all dependencies.
* [curl_chrome110](chrome/curl_chrome110), [curl_chrome124](chrome/curl_chrome124) - Wrapper scripts that launch `curl-impersonate` with the correct flags.
* [curl-impersonate.patch](chrome/patches/curl-impersonate.patch) - The main patch that makes curl use the same TLS extensions as Firefox. Also makes curl compile statically with libnghttp2.
* [boringssl.patch](chrome/patches/boringssl.patch) - The boringssl patch that tweaks boringssl behaviors.
* [win](win) - Scripts for building the Windows version of `curl-impersonate`, which is quite different from `*nix`.
* [zigshim](zigshim) - We use the awesome `zig` toolchain to bring `curl-impersonate` to more archs on Linux. Special thanks to @bjia56 for making it possible.
* [docker](docker) - Debian and alpine dockerfiles for this project.
Other files of interest:
Expand Down
0 chrome/Dockerfile.alpine → docker/alpine.dockerfile
View file
File renamed without changes.
0 chrome/Dockerfile → docker/debian.dockerfile
View file
File renamed without changes.
Loading

0 comments on commit 67828d8

Please sign in to comment.