Distribute binaries, including a drop-in replacement for the shared library

[somini]

Thanks for publishing this.

I see the release process is fairly automated, is it possible to use CI to automatically build binaries of this, and publish them as releases?
I don't have experience in Github Actions, but I do in Gitlab CI.
Both containers take 3 GB, but the binaries themselves are only ~7 MB.

Another interesting build artefact would be to "bake in" the settings in curl_ff95 script in the build itself, and create a regular curl shared library. This could replace with the upstream libcurl file to use the new settings, or hacked around with LD_LIBRARY_PATH.
It would need to be a different build per browser.

Now curl is built in static mode. The library itself is in lib/.libs/libcurl.a.

[lwthiker]

Thanks for the suggestions.

I agree about libcurl, I believe many would find it useful. If we "bake in" all the settings directly it would result in many copies of libcurl, one for each browser version. If we support 10 Chrome versions then we'll have 10 different libcurls. Maybe a better solution would be to add a function to libcurl that configures everything the right way? But then LD_LIBRARY_PATH wouldn't work as you suggested.

I used GitHub Actions for small projects only; I don't know if it supports our heavy builds for free. Worth a try I guess. Would be good to have it to automatically check PRs, together with the new testing infra (#9).

About publishing, I'm not sure whether the binaries would be useful by themselves, as they are compiled for a specific Ubuntu/Debian and have external dependencies. Maybe publishing a .deb/.rpm package? I'm really not a packaging expert though.

[somini]

Maybe a better solution would be to add a function to libcurl that configures everything the right way?

Of course, this is the end game. This is really a stop-gap solution. My usecase is really other applications that link directly to libcurl. Even if libcurl includes these changes, it might not be immediate to update.

About publishing, I'm not sure whether the binaries would be useful by themselves, as they are compiled for a specific Ubuntu/Debian and have external dependencies.

Each distro has it's own package, so the dependencies would be installed automatically. Compiling the curl binaries would still be useful, since most dependencies are statically compiled.
The missing dependencies can be solved by installing the distro curl.

[lwthiker]

I see, so one possibility is to set all the command line parameters and headers in one of libcurl's initialization functions. It could depend on some env var so that we don't need different libcurl compilations for different browser versions (the cipher list, the user agents, etc. are changing every version). Then it might be possible to just replace libcurl for an existing application using LD_LIBRARY_PATH. Worth a try.

[lwthiker]

@somini You can check out #13, it's still WIP but it's in a good shape. I'm curious to know whether it works for you.

[somini]

Sorry for disappearing, this notification got lost in a sea of other notifications, I only had time to get back to this now.
I'm still very much interested.

#13 sounds perfect, I'm still working on building this and test it with a proper application (RSS Bridge, in this case).
I'm trying to build the AUR package and hit a snag: https://aur.archlinux.org/packages/curl-impersonate-firefox#comment-859457

[somini]

Finally got to test this and it very much works as intended, thanks. The AUR packages can be built without errors.

Any progress on getting something like this upstreamed on cURL? I think NSS is being dropped from curl, so once that hits, impersonating firefox is out of the question.

[jlarmstrongiv]

As far as I’m aware, there’s no limit on Github Actions for public repos, so it’d be worthwhile to use the use the CI to automatically build and release binaries.

The only limiting factor with GitHub actions that I have found is Dockerhub pull rate limiting, but I never bothered to setup sign-ins, though even that may not be necessary anymore.

[jlarmstrongiv]

GitHub Actions can also build on different platforms, e.g. MacOS and Windows, which would help solve issues like #24

[lwthiker]

I'm working on a Makefile-based build that will compile curl-impersonate natively, without the Docker containers. Once this is up and working it'll be possible to add GitHub Action to automatically compile to MacOS and Linux and add the binaries to the releases

[lwthiker]

Binaries for Linux & macOS are now automatically built and distributed in the GitHub release page. You can find the latest binaries in release v0.4.1.

[izzues]

Sorry for the dumb question, but am I missing something or is libcurl not being automatically built despite this issue being closed?

[lwthiker]

You are right, only the static binary is currently being built automatically. Do you need a pre-built libcurl in addition?

[momala454]

personally I do for ease of updating

[izzues]

You are right, only the static binary is currently being built automatically. Do you need a pre-built libcurl in addition?

It would be very useful, yes! It's also in the title and description of this issue, hence why I assumed it would have been done.

[lwthiker]

libcurl-impersonate binaries are also added right now to the release page.