Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

buildsystem: support get handlers (archive, file, git) #2529

Merged
merged 9 commits into from Apr 2, 2018

Conversation

@MilhouseVH
Copy link
Contributor

commented Feb 27, 2018

This is a feature that came out of #2517.

It adds support for packages file formats that may be of more use to developers (file/directory), and other distributions (git).

The get_archive handler implements current tar archive download functionality.

The get_file handler supports files and directories using file://<path> urls. If the file is a tar archive then it will be unpacked directly into the build directory (if not a tar archive it will be copied, but the build will most likely fail). Directories will be copied directly into the build directory.

The get_git handler supports git repositories using either git:// urls, or urls that end in .git. Repositories will be cloned into a directory in $SOURCES, and from there copied into the build directory.

The distro option GET_HANDLER_SUPPORT will determine which handlers are supported. By default LibreELEC will support only the "archive" handler. Add "file" or "git" to to enable additional handlers, using either $HOME/.libreelec/options for local development, or the relevant distro options file.

Thanks to @ToKe79 for get_git.

scripts/get Outdated
echo "ERROR: get handler \"${get_handler}\" is not supported, unable to get package $1 - aborting!"
exit 1
else
get_handler="./scripts/get_${get_handler}"

This comment has been minimized.

Copy link
@ToKe79

ToKe79 Feb 28, 2018

Contributor

get_handler="${SCRIPTS}/get_${get_handler}"

This comment has been minimized.

Copy link
@MilhouseVH

MilhouseVH Feb 28, 2018

Author Contributor

Thanks, updated.

@MilhouseVH MilhouseVH force-pushed the MilhouseVH:le90_get_handlers branch 2 times, most recently from a0c969b to 9a50a3c Feb 28, 2018

@MilhouseVH

This comment has been minimized.

Copy link
Contributor Author

commented Mar 1, 2018

@ToKe79 a couple of extra commits have been added after testing from @HiassofT - these should only affect file:// as it was found that when copying a directory the target directory is not being created correctly.

Also, using tar | untar also allows specific files or directories to be excluded during the copy into the build dir - by default .git and .svn will be excluded when copying a file://.

This could also be useful for git repositories created by get_git, for instance if you wanted to exclude certain files/directories from the git repo so that they are not copied into the build dir, but you'd need to manually specify the files to be excluded in the package.mk (eg. PKG_TAR_COPY_OPTS="--exclude .git").

If .git and .svn are needed when copying a file:// then the package.mk should specify PKG_TAR_COPY_OPTS="".

Default excludes (ie. .git, .svn) are only used when PKG_TAR_COPY_OPTS is not already set, so if additional excludes are necessary then the default will not be used and the package.mk would need to set PKG_TAR_COPY_OPTS="--exclude .git --exclude .svn --exclude .temp" etc.

I don't think these changes should have any affect on your git repositories, but can you test/confirm?

@ToKe79

This comment has been minimized.

Copy link
Contributor

commented Mar 1, 2018

@MilhouseVH
TL;DR; As you wrote, this should not impact the git handler at all.

Review: Looking at the changes of config/path, the PKG_TAR_COPY_OPTS are set to default (--exclude .git --exclude .svn) only in case PKG_URL is a file://... url, so this should not impact the get_git handler. (git://... and ...repo.git urls). tar -cf | tar -xf should in such case copy all contents including the required .git folder, unless explicitly specified in package.mk with PKG_TAR_COPY_OPTS. But to make sure, I will test with some packages at my end (I have to rebuild the toolchain, as I had already changes from master compiled).

@ToKe79

This comment has been minimized.

Copy link
Contributor

commented Mar 2, 2018

I confirm that above changes have no conflicts with the git handler.

I added some enhancements to get_git - MilhouseVH#5

For git clones I am not using STAMPS but I am checking URL / commit / branch directly from the cloned folder before downloading (_get_repo_already_downloaded modified).

I also modified the checking of existing folders in $SOURCES/$1 and if current branch does not match $PKG_GIT_CLONE_BRANCH, I check if I can checkout the branch either existing local branch or a remote branch.

Before I reset the head to specific commit I test if such commit exists in the history on the current branch to not reset the head to a commit on different local branch.

I also suppressed the output of the cd -.

@ToKe79

This comment has been minimized.

Copy link
Contributor

commented Mar 2, 2018

Added _get_repo_clean and call it during _get_repo_already_downloaded and before git pull / git checkout to remove local changes.

@MilhouseVH

This comment has been minimized.

Copy link
Contributor Author

commented Mar 22, 2018

Merge me?

MilhouseVH and others added 9 commits Feb 25, 2018
scripts/extract: use tar to copy to correct directory
also add support for custom tar opt, eg to exclude files
or directories when copying

This can be set eg with PKG_TAR_COPY_OPTS="--exclude=.git"
config/path: don't copy .git and .svn dirs when using file:///SOMEDIR
packages can override that by setting PKG_TAR_COPY_OPTS. If this
is set to an empty value everything will be copied.
get_git: fixes and enhancements - fix checking of downloaded repo - c…
…lean local changes in cloned repo - enhance branch handling - add commit checking on the branch

@MilhouseVH MilhouseVH force-pushed the MilhouseVH:le90_get_handlers branch from f97cd06 to fa74600 Apr 2, 2018

@MilhouseVH

This comment has been minimized.

Copy link
Contributor Author

commented Apr 2, 2018

Is there something preventing a merge?

@CvH
CvH approved these changes Apr 2, 2018

@CvH CvH merged commit 42f3a2f into LibreELEC:master Apr 2, 2018

@vpeter4

This comment has been minimized.

Copy link
Contributor

commented on scripts/get_archive in 3d17135 Aug 9, 2019

Typo :)
Cant't get -> Can't get

This comment has been minimized.

Copy link
Contributor Author

replied Aug 9, 2019

Thanks - fixed by 7beb205 in #3745

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.