buildsystem: support get handlers (archive, file, git)

[MilhouseVH]

This is a feature that came out of #2517.

It adds support for packages file formats that may be of more use to developers (file/directory), and other distributions (git).

The get_archive handler implements current tar archive download functionality.

The get_file handler supports files and directories using file://<path> urls. If the file is a tar archive then it will be unpacked directly into the build directory (if not a tar archive it will be copied, but the build will most likely fail). Directories will be copied directly into the build directory.

The get_git handler supports git repositories using either git:// urls, or urls that end in .git. Repositories will be cloned into a directory in $SOURCES, and from there copied into the build directory.

The distro option GET_HANDLER_SUPPORT will determine which handlers are supported. By default LibreELEC will support only the "archive" handler. Add "file" or "git" to to enable additional handlers, using either $HOME/.libreelec/options for local development, or the relevant distro options file.

Thanks to @ToKe79 for get_git.

[ToKe79]

get_handler="${SCRIPTS}/get_${get_handler}"

[MilhouseVH]

Thanks, updated.

[MilhouseVH]

@ToKe79 a couple of extra commits have been added after testing from @HiassofT - these should only affect file:// as it was found that when copying a directory the target directory is not being created correctly.

Also, using tar | untar also allows specific files or directories to be excluded during the copy into the build dir - by default .git and .svn will be excluded when copying a file://.

This could also be useful for git repositories created by get_git, for instance if you wanted to exclude certain files/directories from the git repo so that they are not copied into the build dir, but you'd need to manually specify the files to be excluded in the package.mk (eg. PKG_TAR_COPY_OPTS="--exclude .git").

If .git and .svn are needed when copying a file:// then the package.mk should specify PKG_TAR_COPY_OPTS="".

Default excludes (ie. .git, .svn) are only used when PKG_TAR_COPY_OPTS is not already set, so if additional excludes are necessary then the default will not be used and the package.mk would need to set PKG_TAR_COPY_OPTS="--exclude .git --exclude .svn --exclude .temp" etc.

I don't think these changes should have any affect on your git repositories, but can you test/confirm?

[ToKe79]

@MilhouseVH
TL;DR; As you wrote, this should not impact the git handler at all.

Review: Looking at the changes of config/path, the PKG_TAR_COPY_OPTS are set to default (--exclude .git --exclude .svn) only in case PKG_URL is a file://... url, so this should not impact the get_git handler. (git://... and ...repo.git urls). tar -cf | tar -xf should in such case copy all contents including the required .git folder, unless explicitly specified in package.mk with PKG_TAR_COPY_OPTS. But to make sure, I will test with some packages at my end (I have to rebuild the toolchain, as I had already changes from master compiled).

[ToKe79]

I confirm that above changes have no conflicts with the git handler.

I added some enhancements to get_git - https://github.com/MilhouseVH/LibreELEC.tv/pull/5

For git clones I am not using STAMPS but I am checking URL / commit / branch directly from the cloned folder before downloading (_get_repo_already_downloaded modified).

I also modified the checking of existing folders in $SOURCES/$1 and if current branch does not match $PKG_GIT_CLONE_BRANCH, I check if I can checkout the branch either existing local branch or a remote branch.

Before I reset the head to specific commit I test if such commit exists in the history on the current branch to not reset the head to a commit on different local branch.

I also suppressed the output of the cd -.

[ToKe79]

Added _get_repo_clean and call it during _get_repo_already_downloaded and before git pull / git checkout to remove local changes.

[MilhouseVH]

Merge me?

[MilhouseVH]

Is there something preventing a merge?