Build and packaing clang-tools binaries with this project

[shenxianpeng]

No description provided.

[2bndy5]

There are a couple noteworthy distributions on pypi that facilitate access to LLVM's tooling:

1. clang-format seems reliable and ships with executable binary. The version numbers do correspond to the version of LLVM that was built.
2. libclang which offers an API that can be used similar to clang-tidy. This repo differs from clang in that libclang actually ships with a built binary library (clang relies on the a pre-existing installation).
   • libclang v12 suffered from poor understanding of pypi uploads, and unfortunately cannot be used.

I have been using libclang for a different project (the one generating C++ documentation in sphinx). I haven't tried using clang-format from pip for anything other than latest release, so I'm not sure if there are historic distribution problems that might be detrimental.

[shenxianpeng]

I noticed clang-format when I want to create the cpp-linter-hooks for pre-commit. I want to support both clang-format and clang-tidy, but it only supports clang-format. so I used https://github.com/muttleyxd/clang-tools-static-binaries which provided static binaries of clang-format and clang-tidy.

Keeping the same version of LLVM on PyPI is challenging work because PyPI doesn't support releasing the same version number if the version already be used, so each release must succeed. (You could test on testpypi, but there is no guarantee that you will want to re-release with the same version number.)

Like you mentioned libclang which could support both clang-format and clang-tidy? I might need to take some time to learn about it.

[2bndy5]

I know libclang can be used like clang-tidy, but I'm not sure if it can be used like clang-format. When I parse a C/C++ file with libclang, it produces "diagnostics" which are identical to the diagnostics output from clang-tidy.

I only threw those idea out there as an example of how the binaries could be distributed. I don't know if they are completely useful since our aim is to provide as many clang versions as possible.

[shenxianpeng]

clang-format really inspired me about how to distribute binaries. So far this repo is working fine, no hurry to start this ASAP(welcome to contribute). But if this repo can support both downloading and installing binaries very convenient, it should be welcomed.

[2bndy5]

There is a way to specify extra files (in setup.py) to be included with distributions. However, clang-tidy-14 (of the built binaries) has seg fault problems with Ubuntu 22.02, and the macosx binary of clang-format-14 is over 1GB. If we go this route, then we have to make sure the supported versions of the binaries distributed are actually ok to distribute.

[shenxianpeng]

Do you recommend installing binaries with installing-additional-files? put all install steps here might a 'big' setup.py file. Currently, clang-tidy-14 has seg fault problems, so I want to rebuild it but still in my backlog.

make sure the supported versions of the binaries distributed are actually ok to distribute.

Do you mean there could be legal issues involved?

Any other options? maybe install clang-tools like KyleMayes/install-llvm-action?

[2bndy5]

Do you recommend installing binaries with installing-additional-files?

Yeah, we could bundle all the supported binaries in a distribution and avoid having to download from the release page.

Do you mean there could be legal issues involved?

Kinda. We could do a multi-license when we start distributing the binaries. I really meant that we should make sure the binaries aren't broken or too big.

Any other options? maybe install clang-tools like KyleMayes/install-llvm-action?

That action requires node.js and some npm packages installed. It works fine for github actions because github uses node.js as its driving force, thus the different type of actions used for non-node.js based actions (like ours).

[shenxianpeng]

Yeah, we could bundle all the supported binaries in distribution and avoid having to download from the release page.

Like a file.zip include all binaries? I also want to use this project for cpp-linter-hooks and download specific versions of binaries at runtime. if the distribution includes all binaries will take more time.

I really meant that we should make sure the binaries aren't broken or too big.

So distribute ourselves and use the CI tests can make sure it always works.

Got it. I mean to install like KyleMayes/install-llvm-action but we used Python program. but it should take more time than downloading binaries. so I prefer to distribute the binaries

[2bndy5]

if the distribution includes all binaries will take more time.

We could make it so only the binaries for the installing platform are distributed. We could also drop older versions of clang tools that aren't supported anymore (like x < v10). However, you can't optionally distribute the binaries. It would take 2 separate pypi packages (1 pkg without the binaries & 1 pkg with the binaries), and in that case, you may as well download the binaries on demand anyway.

[shenxianpeng]

Your revisions are great enough (download binaries from the GitHub release page), do we still need to install binaries with installing-additional-files?

[2bndy5]

No, I think the current approach is best.

[2bndy5]

I've been trying to see what can be done for the v14 builds, and I've got to ask:

Is there any problem with hosting our own repo that contains the released binaries built by LLVM devs?

I'm under the impression that this is what the docker image is doing already but for Linux only. It might be easier than trying to build our own binaries (I forked the clang-tools-binaries repo). Each build takes about 4 hours (even with clang-query and versions <= 8 excluded from the CI).

[shenxianpeng]

Is there any problem with hosting our own repo that contains the released binaries built by LLVM devs?

See from llvm project license below, follow their license agreement I think it's OK to release binaries.

---

The “Apache 2.0 License with LLVM exceptions” allows you to:

• freely download and use LLVM (in whole or in part) for personal, internal, or commercial purposes.
• include LLVM in packages or distributions you create.
• combine LLVM with code licensed under every other major open source license (including BSD, MIT, GPLv2, GPLv3…).
• make changes to LLVM code without being required to contribute it back to the project - contributions are appreciated though!

However, it imposes these limitations on you:

• You must retain the copyright notice if you redistribute LLVM: You cannot strip the copyright headers off or replace them with your own.
• Binaries that include LLVM must reproduce the copyright notice (e.g. in an included README file or in an “About” box), unless the LLVM code was added as a by-product of compilation. For example, if an LLVM runtime library like compiler_rt or libc++ was automatically included into your application by the compiler, you do not need to attribute it.
• You can’t use our names to promote your products (LLVM derived or not) - though you can make truthful statements about your use of the LLVM code, without implying our sponsorship.
• There’s no warranty on LLVM at all.

[2bndy5]

Binaries that include LLVM must reproduce the copyright notice (e.g. in an included README file or in an “About” box), unless the LLVM code was added as a by-product of compilation. For example, if an LLVM runtime library like compiler_rt or libc++ was automatically included into your application by the compiler, you do not need to attribute it.

I think this one is the only one that would apply to my idea. We could simply add a COPY.txt to our repo that would host the released binaries. So, COPY.txt would contain the license information and maybe a link to the LLVM project source.

The alternative is to download the LLVM release asset directly and extract only clang-tidy/format from it (the exe files for windows can be treated as zip archives). However, those assets are around 500 MB each, and that might slow down this pkg's executable script (I wish there was a simple way to show a download progress bar).

[shenxianpeng]

It's great to show the download progress

[2bndy5]

I recently used the python std lib curses (a c-extension wrapper lib), but that's only natively available on linux. We'd have to either draw the bar manually or use rich (which is a bit bloated for our scenario). This stack overflow answer just shows a textual percentage... I'm getting off topic again.

[shenxianpeng]

Let the good idea be noted #18

[2bndy5]

Turns out the releases from LLVM on github are dynamically linked. So, I'm returning focus to fixing the clang-tools-static-binaries repo's workflow. I already was able to fix clang-tidy-14 on linux by using Ubuntu-22.04 (which is still in early release on actions/runner-images.

However, I don't own a Apple PC to test build the clang-tidy-14 locally. Anything I try will have to be done on the github CI (which takes around 4 hours if I don't disable all other matrix contexts). So, far I've tried building the latest patch release (14.0.6) and changed the CMAKE_OSX_DEPLOYMENT_TARGET=11 from the deprecated 10.15 value.

[2bndy5]

I finally found the LLVM doc that details the CMake variables:
https://llvm.org/docs/CMake.html#options-and-variables

[shenxianpeng]

Good to know clang-tidy-14 can be fixed on Ubuntu. I don't have Mac either. Only keeping version 14 for testing with GitHub CI can save much time.

[BurningEnlightenment]

Any news here? I recently switched to clang-15 for CI linting by installing dynamically linked binaries from the llvm apt repository. But that setup seems a little bit brittle due to the fact that I need to specify the full version for the linter action (otherwise it tries to download the broken static binaries), but OTOH I can't specify the exact version to be installed via apt-get.

[…]
      - name: "Install clang-tools-15"
        run: |
          sudo cp ${{ github.workspace }}/tools/llvm-snapshot.gpg.key.asc /etc/apt/trusted.gpg.d/llvm-snapshot.asc
          sudo add-apt-repository "deb http://apt.llvm.org/jammy/ llvm-toolchain-jammy-15 main"
          sudo apt-get -qq update
          sudo apt-get -qq -y install clang-15 clang-tidy-15 clang-format-15
[…]
      - uses: cpp-linter/cpp-linter-action@v2
        name: Lint with clang-format and clang-tidy
        id: linter
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        with:
            version: 15.0.7
            style: file # use .clang-format settings
            tidy-checks: '' # use .clang-tidy settings
            extensions: cpp,hpp
            ignore: '.github|build|external'
            files-changed-only: true
            database: build/x64-linux-ci-lint

[2bndy5]

This issue is about distributing statically linked binaries with pypi, but it has gotten stale because we found the idea rather unfeasible. Just to summarize the ideas discussed:

• distributing/using statically linked binaries is discouraged because it expects a certain version of C/C++ std libs.
• other solutions (like the pypi pkgs clang-format or libclang) don't provide a comprehensive history to offer whatever version users may need.
  • using libclang instead of clang-tidy would require a serious overhaul of the cpp-linter pkg's behavior, and it still doesn't offer all versions of clang.

Additionally, from another issue on this repo, we had decided to require the full version number to specify a minor or patched release of a major version (v12 is v12.0.0 but not v12.0.1). Personally, I've grown to find apt packages unreliable when specific versions are required. Instead, you could use another action (like KyleMayes/install-llvm-action) prior to using cpp-linter; remember you can also pass the path to the bin folder as the version option's value to cpp-linter-action.

[LecrisUT]

How about using scikit-build-core? clang-format pypi package uses that. Couple that with synchronizing the llvm version with this package version.

[2bndy5]

I'm not a fan of syncing this package's version number with the LLVM version number because this pkg is meant to download and install a specific version of clang-tools. This means that a user should be able to install clang-*-12 and/or clang-*-14 using the same version of clang-tools-pip pkg.

To put it simply, this pkg isn't meant to be a python port of clang-tools. It is only meant for downloading and installing clang-tools binaries. I will concede that this pkg's name should've been chosen to better reflect that distinction (I didn't pick the name).

[shenxianpeng]

We have already forked and built clang-tools binaries by ourselves in https://github.com/cpp-linter/clang-tools-static-binaries, In the meantime

• older versions of clang-tools, including 6, 5, 4, and 3.9 are no longer supported
• also fixed the clang-tidy 14 segfault muttleyxd/clang-tools-static-binaries#18 in the clang-tidy v14+ version on Ubuntu
• add clang-apply-replacements binaries per the upstream user's request.

So I think this ticket can be closed and please feel free to reopen it if needed.