tc-build supports building from tarballs

[dileks]

Does tc-build supports building from tarballs?

If yes , where do I have to place/unpack the tarballs speaking of the directory structure?

Currently, I see version 10.0.0-rc6 and wanted to do a stage1 build with LLVM, Clang and LLD tarballs.

Thanks to Universe the Clang tarball has no more cfe as prefix.

[1] https://prereleases.llvm.org/10.0.0/
[2] https://prereleases.llvm.org/10.0.0/rc6/llvm-10.0.0rc6.src.tar.xz
[3] https://prereleases.llvm.org/10.0.0/rc6/clang-10.0.0rc6.src.tar.xz
[4] https://prereleases.llvm.org/10.0.0/rc6/lld-10.0.0rc6.src.tar.xz

[nathanchance]

No, it does not. Just pass in --branch llvmorg-10.0.0-rc6 to check out the git repository that those tarballs were generated from.

[dileks]

Can you tell me how big such a git-checkout is?
Currently, I am using UMTS/LTE to connect to the Internet, so transferred data volume matters for me.

[nathanchance]

A full checkout is over a GB but a shallow clone (which can be done via --shallow-clone) is 123.25 MiB.

Alternatively, this works for me (assuming llvm-project does not already exist):

$ mkdir -p llvm-project

$ cd llvm-project

$ for PROJECT in llvm clang lld; do mkdir -p "${PROJECT}" && curl -LSs https://github.com/llvm/llvm-project/releases/download/llvmorg-10.0.0/"${PROJECT}"-10.0.0.src.tar.xz | tar -C "${PROJECT}" --strip-components=1 -xJf -; done

$ cd ..

$ ./build-llvm.py --build-stage1-only --no-update --projects "clang;lld"

[dileks]

Cool, LLVM version 10.0.0 was released 5 hours ago :-).

Did not know of --shallow-clone git-clone option, Thanks.

Will try later - both solutions.

[1] https://github.com/llvm/llvm-project/releases/tag/llvmorg-10.0.0

[dileks]

Sizes, please:

34.1 MB llvm-10.0.0.src.tar.xz
13.5 MB clang-10.0.0.src.tar.xz
1.09 MB lld-10.0.0.src.tar.xz

You happen to know what is included here?

80.5 MB llvm-project-10.0.0.tar.xz

[nathanchance]

llvm-project is https://github.com/llvm/llvm-project/tree/llvmorg-10.0.0 compressed.

[dileks]

So, it is simpler and "cleaner" to download llvm-project-10.0.0.tar.xz which includes all projects means Clang and LLD what I want.

[nathanchance]

Yes.

mkdir -p llvm-project && curl -LSs https://github.com/llvm/llvm-project/releases/download/llvmorg-10.0.0/llvm-project-10.0.0.tar.xz | tar -C llvm-project --strip-components=1 -xJf -

then just simply adding --no-update to your regular build-llvm.py invocation.

[dileks]

Cool.

The new llvm-project Git infrastructure offers a lot of new opportunities.

I was thinking of compiler regression testing with final releases.

What about offering a "tarball download" option (with setting "--no-update")?

[1] https://github.com/llvm/llvm-project/releases/download/llvmorg-9.0.1/llvm-project-9.0.1.tar.xz
[2] https://github.com/llvm/llvm-project/releases/download/llvmorg-10.0.0/llvm-project-10.0.0.tar.xz

[nathanchance]

I'd rather not complicate the script with another download option. --shallow-clone is available for most resource conscious scenarios (be it time or bandwidth). If that does not work, I feel that manually managing llvm-project and passing in --no-update is a completely reasonable workflow. I don't want the script to do everything from a source management perspective. I might be able to document this somewhere (README, I guess?).

[dileks]

Yeah, you are right - KISS.

Can you comment on the --shallow-clone option - how to do that?
My git version 2.25.1 does not know of git clone --shallow-clone.

[nathanchance]

Add --shallow-clone to ./build-llvm.py for it to take care of it; otherwise, clone llvm-project with --depth=1.

[dileks]

OK, I see this is a feature of tc-build not git.

Please, document this in README.md.

[nathanchance]

There is a -h/--help option for clear documentation of all of these flags, which is mentioned in the README.

[dileks]

I meant feel free to document your comments in #75 (comment) in README.md.

[dileks]

Thanks @nathanchance for your proposal.

I like documentation with examples especially in man-pages.

Please add an example - something like this:

Example: Use llvm-project tarball version 10.0.0 building Clang and LLD (stage1-only build)

$ cd /path/to/tc-build
$ curl -LSs https://github.com/llvm/llvm-project/releases/download/llvmorg-10.0.0/llvm-project-10.0.0.tar.xz | tar -C llvm-project --strip-components=1 -xf -
$ cd ..
$ ./build-llvm.py --build-stage1-only --projects "clang;lld" --no-update

Place the block where you think it's useful (or do a separate Example paragraph in the README).

UPDATE: Move --no-update at the end.
UPDATE-2: Simplify tar decompress-option -xf is enough.

[dileks]

News: Debian/unstable now offers llvm-toolchain-10 version 1:10.0.0-1.

[1] https://packages.debian.org/sid/clang-10

[sylvestre]

News: Debian/unstable now offers llvm-toolchain-10 version 1:10.0.0-1.

[1] https://packages.debian.org/sid/clang-10

And always happy to help you folks if needed :)

[dileks]

@sylvestre

( Might be Off-Topic )
Is "[PATCH] Fix integration of pass plugins with llvm dylib" needed/useful when building a selfmade toolchain?

[1] https://sources.debian.org/src/llvm-toolchain-10/1:10.0.0-1/debian/patches/d21664cce1db8debe2528f36b1fbd2b8af9c9401.patch

[nathanchance]

Thanks @nathanchance for your proposal.

I like documentation with examples especially in man-pages.

Please add an example - something like this:

Example: Use llvm-project tarball version 10.0.0 building Clang and LLD (stage1-only build)

$ cd /path/to/tc-build
$ curl -LSs https://github.com/llvm/llvm-project/releases/download/llvmorg-10.0.0/llvm-project-10.0.0.tar.xz | tar -C llvm-project --strip-components=1 -xJf -
$ cd ..
$ ./build-llvm.py --build-stage1-only --projects "clang;lld" --no-update

Place the block where you think it's useful (or do a separate Example paragraph in the README).

UPDATE: Move --no-update at the end.

I would rather keep the README as uncluttered as possible, especially since this isn't really something that I believe people should be doing regularly. I put a lot of work into making sure the repo part of the script works, I would prefer most people use that than doing everything themselves and if they cannot understand what I am describing there, they probably should not be doing that anyways.

I've added a link to your post though so that people can refer back to it if need be.

[sylvestre]

Is "[PATCH] Fix integration of pass plugins with llvm dylib" needed/useful when building a selfmade toolchain?

I don't think it is. It has been cherry-picked to fix a link with for Mesa (when polly is enabled)

[dileks]

@nathanchance
Thanks for merging and linking to my comment.

@sylvestre
Thanks for the heads-up.

[mykyglazov]

Hi, which command i need to type for instalation the 9.3 version of llvm? OS - Ubuntu. Thx

[nathanchance]

9.3? Are you talking about 9.0.3 from AOSP or 9.0.1 from upstream LLVM?