Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Distribute binaries, including a drop-in replacement for the shared library #8

Closed
somini opened this issue Feb 21, 2022 · 15 comments
Closed

Comments

@somini
Copy link

somini commented Feb 21, 2022

Thanks for publishing this.

I see the release process is fairly automated, is it possible to use CI to automatically build binaries of this, and publish them as releases?
I don't have experience in Github Actions, but I do in Gitlab CI.
Both containers take 3 GB, but the binaries themselves are only ~7 MB.

Another interesting build artefact would be to "bake in" the settings in curl_ff95 script in the build itself, and create a regular curl shared library. This could replace with the upstream libcurl file to use the new settings, or hacked around with LD_LIBRARY_PATH.
It would need to be a different build per browser.

Now curl is built in static mode. The library itself is in lib/.libs/libcurl.a.

@lwthiker
Copy link
Owner

Thanks for the suggestions.

I agree about libcurl, I believe many would find it useful. If we "bake in" all the settings directly it would result in many copies of libcurl, one for each browser version. If we support 10 Chrome versions then we'll have 10 different libcurls. Maybe a better solution would be to add a function to libcurl that configures everything the right way? But then LD_LIBRARY_PATH wouldn't work as you suggested.

I used GitHub Actions for small projects only; I don't know if it supports our heavy builds for free. Worth a try I guess. Would be good to have it to automatically check PRs, together with the new testing infra (#9).

About publishing, I'm not sure whether the binaries would be useful by themselves, as they are compiled for a specific Ubuntu/Debian and have external dependencies. Maybe publishing a .deb/.rpm package? I'm really not a packaging expert though.

@somini
Copy link
Author

somini commented Feb 22, 2022

Maybe a better solution would be to add a function to libcurl that configures everything the right way?

Of course, this is the end game. This is really a stop-gap solution. My usecase is really other applications that link directly to libcurl. Even if libcurl includes these changes, it might not be immediate to update.

About publishing, I'm not sure whether the binaries would be useful by themselves, as they are compiled for a specific Ubuntu/Debian and have external dependencies.

Each distro has it's own package, so the dependencies would be installed automatically. Compiling the curl binaries would still be useful, since most dependencies are statically compiled.
The missing dependencies can be solved by installing the distro curl.

@lwthiker
Copy link
Owner

I see, so one possibility is to set all the command line parameters and headers in one of libcurl's initialization functions. It could depend on some env var so that we don't need different libcurl compilations for different browser versions (the cipher list, the user agents, etc. are changing every version). Then it might be possible to just replace libcurl for an existing application using LD_LIBRARY_PATH. Worth a try.

@lwthiker
Copy link
Owner

@somini You can check out #13, it's still WIP but it's in a good shape. I'm curious to know whether it works for you.

@somini
Copy link
Author

somini commented Apr 2, 2022

Sorry for disappearing, this notification got lost in a sea of other notifications, I only had time to get back to this now.
I'm still very much interested.

#13 sounds perfect, I'm still working on building this and test it with a proper application (RSS Bridge, in this case).
I'm trying to build the AUR package and hit a snag: https://aur.archlinux.org/packages/curl-impersonate-firefox#comment-859457

@somini
Copy link
Author

somini commented Apr 3, 2022

Finally got to test this and it very much works as intended, thanks. The AUR packages can be built without errors.

Any progress on getting something like this upstreamed on cURL? I think NSS is being dropped from curl, so once that hits, impersonating firefox is out of the question.

@jlarmstrongiv
Copy link

jlarmstrongiv commented Apr 4, 2022

As far as I’m aware, there’s no limit on Github Actions for public repos, so it’d be worthwhile to use the use the CI to automatically build and release binaries.

The only limiting factor with GitHub actions that I have found is Dockerhub pull rate limiting, but I never bothered to setup sign-ins, though even that may not be necessary anymore.

@jlarmstrongiv
Copy link

GitHub Actions can also build on different platforms, e.g. MacOS and Windows, which would help solve issues like #24

@lwthiker
Copy link
Owner

lwthiker commented Apr 6, 2022

I'm working on a Makefile-based build that will compile curl-impersonate natively, without the Docker containers. Once this is up and working it'll be possible to add GitHub Action to automatically compile to MacOS and Linux and add the binaries to the releases

@lwthiker
Copy link
Owner

Binaries for Linux & macOS are now automatically built and distributed in the GitHub release page. You can find the latest binaries in release v0.4.1.

@izzues
Copy link
Contributor

izzues commented May 22, 2022

Sorry for the dumb question, but am I missing something or is libcurl not being automatically built despite this issue being closed?

@lwthiker
Copy link
Owner

lwthiker commented May 23, 2022

You are right, only the static binary is currently being built automatically. Do you need a pre-built libcurl in addition?

@momala454
Copy link

personally I do for ease of updating

@izzues
Copy link
Contributor

izzues commented May 23, 2022

You are right, only the static binary is currently being built automatically. Do you need a pre-built libcurl in addition?

It would be very useful, yes! It's also in the title and description of this issue, hence why I assumed it would have been done.

@lwthiker lwthiker reopened this May 23, 2022
@lwthiker
Copy link
Owner

lwthiker commented Jun 3, 2022

libcurl-impersonate binaries are also added right now to the release page.

@lwthiker lwthiker closed this as completed Jun 3, 2022
kianmeng pushed a commit to kianmeng/curl-impersonate that referenced this issue Aug 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants