Add crashpad [Draft] [Help Needed] #5419

ericriff · 2021-05-03T22:18:27Z

Specify library name and version: crashpad/1.0

This is a draft PR to get some help on a recipe for google crashpad. It is a missing requirement of sentry-native.
This is heavily based on https://github.com/bincrafters/community/blob/main/recipes/crashpad.

I also have a draft PR for Sentry.io's fork of crashpad, which has CMake Support. #5432
Packaging the fork is way easier, but it also has its limitations.

The challenges I'm currently facing are:

Getting the sources.
The official method relies on depot-tools using gclient or sync. This pulls the sources and also the submodules needed to build. If I understand correctly this is a conan antipattern since the submodules should be different packages, right? Even if we move forward with this approach, can the commit hash be listed in conandata.yml (instead of the sources link)?
There is a github mirror that would allow us to do a more traditional conan source pull, listing a github link, a sha556 and then using tools.get(). The problem with this is that it doesn't pull the submodules.
This library uses google's custom ninja files generator called gn. There is work being done right now to package it.
Passing profile configs to the build system.
The bincrafters recipe had this snippet which picks up env variables like CXX, CC, CXXFLAGS and translate them to what the build system understand. This only works if you have this configured in the [env] section of the profile, otherwise they get completely ignored.

    def _set_env_arg(self, args, envvar, gnvar):
        val = os.getenv(envvar)
        if val:
            args += [ "%s=\\\"%s\\\"" % (gnvar, val) ]
 ...
 
    self._set_env_arg(args, "CC",       "custom_cc")
    self._set_env_arg(args, "CXX",      "custom_cxx")
    self._set_env_arg(args, "CFLAGS",   "extra_cflags_c")
    self._set_env_arg(args, "CFLAGS",   "extra_cflags_objc")
    self._set_env_arg(args, "CXXFLAGS", "extra_cflags_cc")
    self._set_env_arg(args, "CXXFLAGS", "extra_cflags_objcc")
    self._set_env_arg(args, "LDFLAGS",  "extra_ldflags")

For example I build with a profile that sets compiler=gcc, compiler.version=8, compiler.libcxx=libstdc++11, but during build clang gets used instead.

What would be the best way to parse all this configurations?

No 32 bits arm support on mini_crhomium
Consuming dependencies as conan packages is not straight forward.

As discused here #5417 (comment) there is a fork of this library which has CMake support. It would be an option to ditch this all together and use those sources for the conan package.

I know there are a lot of smaller problems on this recipe (like the version not being the latest or not starting with cci.), but I would rather focus on the main issues first so please don't comment on those now so we can keep it cleaner :)

I've read the guidelines for contributing.
I've followed the PEP8 style guides for Python code in the recipes.
I've used the latest Conan client version.
I've tried at least one configuration locally with the
conan-center hook activated.

madebr · 2021-05-04T00:34:11Z

I think you can build it without the need for gclient or sync.
It needs python2 though.

git clone gn + build gn: https://gn.googlesource.com/gn/ (conan needs a recipe that packages this executable as a tool)
git clone crashpad
git clone mini_chromium into the third_party\mini_chromium subdirectory of crashpad
run gn gen out/Default in the crashpad folder

This fails on my Windows box because it needs python2.
My error is:

  File "C:\Users\maarten\source\repos\crashpad\third_party\mini_chromium\mini_chromium\build\win_helper.py", line 145
    print line
          ^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print(line)?
ERROR at //third_party/mini_chromium/mini_chromium/build/config/BUILD.gn:568:20: Script returned non-zero exit code.
  toolchain_data = exec_script(helper_path,
                   ^----------
Current dir: C:/Users/maarten/source/repos/crashpad/out/Default/
Command: C:/Python39/python.exe C:/Users/maarten/source/repos/crashpad/third_party/mini_chromium/mini_chromium/build/win_helper.py get-visual-studio-data C:/Users/maarten/source/repos/crashpad/out/Default C:/Users/maarten/source/repos/crashpad/third_party/mini_chromium/mini_chromium/build/config/<autodetect>
Returned 1.
See //build/test.gni:43:7: which caused the file to be included.
      executable(target_name) {

ericriff · 2021-05-04T12:47:30Z

gn is part of depot_tools which is already on Conan so no big deal there.
The problem with getting the sources with git clone is that you can't use conandata.yaml and I think the hooks makes in mandatory. Would it be OK if I use the github mirror for the official sources and then git clone the submodules I need? I think it would be minichromium and lss (single header).
In my opinion those should be conan packages, but I don't think conan can model what this build system needs (the sources and not the artifacts).

madebr · 2021-05-04T12:58:37Z

It looks like it is possible to download certain commits as tarballs.
See https://chromium.googlesource.com/crashpad/crashpad/+archive/b8d3be9b78071f98ad82ff65aa3df7761c05bfc8.tar.gz
Because crashpad needs mini_chromium checked out in a subdirectory, I think you need to also download its tarball in the third_party\mini_chromium subdirectory.

ericriff · 2021-05-04T13:01:28Z

I'll look into it.
It is similar to what I was planning to do but downloading the tarball from the github mirror (not the fork).

ericriff · 2021-05-04T17:46:45Z

Downloading the sources as a taball doesn't work since gn will not work unless you're on a git checkout. I've tried your link and also the one from github.

----Running------
> gn gen out/Conan --args="is_debug=false target_cpu=\"x64\" custom_conan_compiler_args_file=\"@/home/eric/.conan/data/crashpad/cci.20210429/asd/asd/build/b80d46004713aa37d6a90b42e2a326a056a237b5/conanbuildinfo.args\" custom_cxx_is_gcc=true"
-----------------
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
gn.py: Could not find checkout in any parent of the current path.
This must be run inside a checkout.

The same happens if you run gn on a random folder. It wants to be on a git checkout

$ gn --version      
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
gn.py: Could not find checkout in any parent of the current path.
This must be run inside a checkout.

🤔

SSE4 · 2021-05-04T17:55:28Z

this seems to be a (shell script/python) wrapper around gn executable. as building gn from sources results in valid executable which doesn't require checkout.

ericriff · 2021-05-04T18:37:51Z

Looking into gn sources it seems like it would also be a pain to build, it doesn't use a standard build system.
It uses a python script that generates ninja files.
I guess using sentry's fork of crashpad will be way easier.

…sn't build, needs gn]

madebr · 2021-05-04T22:01:52Z

It looks like the google git tarballs are created dynamically and are not reproducible (their hashes always differ).
See e.g. https://gn.googlesource.com/gn/+archive/6771ce569fb4803dad7a427aa2e2c23e960b917e.tar.gz

I don't find an up-to-date github fork for gn.

madebr · 2021-05-04T22:42:05Z

I'm looking into building gn.
build/gn.py has the following options:

  --platform=PLATFORM   target platform (linux/darwin/mingw/msys/msvc/aix/fuch
                        sia/freebsd/netbsd/openbsd/haiku/solaris)
  --host=HOST           host platform (linux/darwin/mingw/msys/msvc/aix/fuchsi
                        a/freebsd/netbsd/openbsd/haiku/solaris)

What does target platform mean in this context?

Does a gn build only support one platform?
e.g. when I build gn to run on Windows (=host).
Is that gn binary only able to build for one platform (=target)?

ericriff · 2021-05-04T23:38:23Z

I have no idea, I have no experience with gn at all. I just need crashpad as a backend for sentry-native.
I have a draft of crashpad using sentry's own fork, which is what they use internally. It seems to work as-is with very little work involved.
It is worth considering using it since building it and handling its dependencies (openSSL and zlib) is straightforward on CMake.

EDIT: See draft PR for getsentry/crashpad here #5432

madebr · 2021-05-05T00:45:24Z

I've created an initial gn recipe at #5433

…les checkouts for the rest

conan-center-bot · 2021-05-06T18:30:03Z

Failure in build 3 (eb45f87410e8398045210b6f1d48b80869cd4db9):

crashpad/cci.20210429@:
CI failed to create some packages (All logs)

Logs for packageID f8bda7f0751e4bc3beaa6c3b2eb02d455291c8a2:

[settings]
arch=x86_64
arch_build=x86_64
build_type=Release
compiler=apple-clang
compiler.libcxx=libc++
compiler.version=10.0
os=Macos
os_build=Macos

********************************************************************************
conan install crashpad/cci.20210429@ --profile=/Users/jenkins/w/BuildSingleReference/11107/3703b487-cbf6-4164-9d2c-02c3a849cbb2/profile.txt --build=crashpad
********************************************************************************
Configuration:
[settings]
arch=x86_64
arch_build=x86_64
build_type=Release
compiler=apple-clang
compiler.libcxx=libc++
compiler.version=10.0
os=Macos
os_build=Macos
[options]
[build_requires]
[env]

crashpad/cci.20210429: Forced build from source
gn/cci.20210429: Not found in local cache, looking in remotes...
gn/cci.20210429: Trying with 'conan-upstream'...
gn/cci.20210429: Trying with 'conan-center'...
ERROR: Failed requirement 'gn/cci.20210429' from 'crashpad/cci.20210429'
ERROR: Unable to find 'gn/cci.20210429' in remotes

^{Note: To save resources, CI tries to finish as soon as an error is found. For this reason you might find that not all the references have been launched or not all the configurations for a given reference. Also, take into account that we cannot guarantee the order of execution as it depends on CI workload and workers availability.}

madebr · 2021-05-06T18:53:34Z

I've pushed a WIP crashpad recipe at https://github.com/madebr/conan-center-index/tree/sentry_wip
It is able to build the client and minidump library (tested with gcc on Linux).
TODO is:

shared/static
packaging (the recipe only builds, does not install)
is static libcurl possible?
the recipe uses embedded mini_chromium and linux-syscall-support, not packages provided by conan. It would be nice to be able to completely switch to conan packages.

ericriff · 2021-05-06T19:13:56Z

I've pushed a WIP crashpad recipe at https://github.com/madebr/conan-center-index/tree/sentry_wip
It is able to build the client and minidump library (tested with gcc on Linux).
TODO is:

shared/static

packaging (the recipe only builds, does not install)

is static libcurl possible?

the recipe uses embedded mini_chromium and linux-syscall-support, not packages provided by conan. It would be nice to be able to completely switch to conan packages.

I've been working on the CMake fork, the draft is on the description if this PR
It is way easier to build with conan.
I think that the package layout will be up to us since crashpad doesn't install.

madebr · 2021-05-06T21:13:29Z

The crashpad recipe at https://github.com/madebr/conan-center-index/tree/sentry_wip is now fully working on Linux (gcc-static).
I don't think I can make shared builds working.
I've re-used the test case of your sentry-crashpad recipe.

ericriff · 2021-05-06T22:20:28Z

I'll do some testing and let you know how it goes.
To keep in mind:

The CMake version explicitly depends on zlib and openSSL on Linux and Android. Does the official sources avoid this somehow?
I've encounter issues building if glibc < 2.27. Apparently crashpad generates a compat library that the bincrafters recipe links only if we're using an older glibc, which makes it work. I don't see any mention of this on your recipe (I improved glibc support in my recipe, but it still not perfect)
Have you tried crosscompiling?
Sentry's fork hardcodes every library to STATIC, maybe there is a reason for it.

ghost · 2021-05-10T21:11:45Z

I detected other pull requests that are modifying crashpad/all recipe:

Crashpad recipe #5511

This message is automatically generated by https://github.com/ericLemanissier/conan-center-conflicting-prs so don't hesitate to report issues/improvements there.

ericriff added 7 commits May 3, 2021 17:29

Import bincafters recipe

003a623

Use conan-index depot-tools

5e5f731

Run test_package only if not cross building

0347714

Patch sources on build() method

ae462c6

Do not export license

51c730e

Declare cmake_minimum_required() earlier

8285488

Make url point to conan-index-center

3b0a1c2

This comment has been minimized.

Sign in to view

conan-center-bot added the Failed label May 3, 2021

jgsogo added the help wanted Need some help to be finish label May 4, 2021

Bump version to latest, pull sources from github, use properties [doe…

513f1eb

…sn't build, needs gn]

ericriff mentioned this pull request May 5, 2021

[Draft][Feedback needed] getsentry-crashpad #5432

Closed

4 tasks

ericriff added 6 commits May 6, 2021 14:42

Use gn package instead of depto_tools

ff31e6f

Correctly list dl and ptrehad as system libs

82da6df

Replace gclient calls with a tarball for the base sources and submodu…

b84b619

…les checkouts for the rest

Improve glibc version detection

aafe04b

Install missing header

640c3d4

Code cleanup

eb45f87

This comment has been minimized.

Sign in to view

This was referenced May 10, 2021

[service] Potentially conflicting recipes #3571

Closed

Crashpad recipe #5511

Merged

ericriff closed this May 10, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add crashpad [Draft] [Help Needed] #5419

Add crashpad [Draft] [Help Needed] #5419

ericriff commented May 3, 2021 •

edited

Loading

This comment has been minimized.

madebr commented May 4, 2021

ericriff commented May 4, 2021

madebr commented May 4, 2021 •

edited

Loading

ericriff commented May 4, 2021

ericriff commented May 4, 2021 •

edited

Loading

SSE4 commented May 4, 2021

ericriff commented May 4, 2021

madebr commented May 4, 2021

madebr commented May 4, 2021

ericriff commented May 4, 2021 •

edited

Loading

madebr commented May 5, 2021

This comment has been minimized.

conan-center-bot commented May 6, 2021

madebr commented May 6, 2021 •

edited

Loading

ericriff commented May 6, 2021

madebr commented May 6, 2021

ericriff commented May 6, 2021 •

edited

Loading

ghost commented May 10, 2021

Add crashpad [Draft] [Help Needed] #5419

Add crashpad [Draft] [Help Needed] #5419

Conversation

ericriff commented May 3, 2021 • edited Loading

This comment has been minimized.

madebr commented May 4, 2021

ericriff commented May 4, 2021

madebr commented May 4, 2021 • edited Loading

ericriff commented May 4, 2021

ericriff commented May 4, 2021 • edited Loading

SSE4 commented May 4, 2021

ericriff commented May 4, 2021

madebr commented May 4, 2021

madebr commented May 4, 2021

ericriff commented May 4, 2021 • edited Loading

madebr commented May 5, 2021

This comment has been minimized.

conan-center-bot commented May 6, 2021

madebr commented May 6, 2021 • edited Loading

ericriff commented May 6, 2021

madebr commented May 6, 2021

ericriff commented May 6, 2021 • edited Loading

ghost commented May 10, 2021

ericriff commented May 3, 2021 •

edited

Loading

madebr commented May 4, 2021 •

edited

Loading

ericriff commented May 4, 2021 •

edited

Loading

ericriff commented May 4, 2021 •

edited

Loading

madebr commented May 6, 2021 •

edited

Loading

ericriff commented May 6, 2021 •

edited

Loading