New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remote package management #69

Open
uael opened this Issue Apr 10, 2017 · 84 comments

Comments

Projects
None yet
4 participants
@uael
Copy link

uael commented Apr 10, 2017

A package manager or package management system is a collection of software tools that automates the process of installing, upgrading, configuring, and removing computer programs for a computer's operating system in a consistent manner.

A package manager deals with packages, distributions of software and data in archive files. Packages contain metadata, such as the software's name, description of its purpose, version number, vendor, checksum, and a list of dependencies necessary for the software to run properly. Upon installation, metadata is stored in a local package database. Package managers typically maintain a database of software dependencies and version information to prevent software mismatches and missing prerequisites. They work closely with software repositories, binary repository managers, and app stores.

Source: package manager

Remote package management roadmap:

  • detect tools and environment and wrap them
    • git
    • curl
    • wget
    • unzip/tar/7z/gzip
    • ping (test speed)
  • Extractor wrapper
    • tar
    • unzip
    • gzip
    • 7z
  • Cache management
  • semver
    • parse
    • satisfies
    • select
    • resolve
  • Multi language
    • asm/c/c++
    • objc/c++
    • swift
    • rust
    • golang
    • dlang
  • Multi platform
    • linux
    • macOS
    • windows
      • git
      • downloader (http[s], ftp ..)
      • extractor (zip, tar.gz, 7z .. )
  • Command line tools
    • require task
      • install
        • parse require info
        • parse and load packages from repositories
        • download package
          • optimization: select the fastest url if exists multiple urls
        • extract package
        • build package
          • xmake.lua
            • linux/macos
            • windows
          • makefile
            • linux/macos
            • windows
          • configure/make
            • linux/macos
          • cmakelists.txt
            • linux/macos
            • windows (in progress ..)
          • *.vcproj
            • windows
        • install package
        • install basic dependent tools (git, curl, unzip ...)
      • search
      • clear
      • remove
      • list
    • repo task
      • add repository
      • remove repository
      • list all repositories
      • clear all repositories and cache
      • global or local repositories
  • xmake lua api
    • add_requires: add package require info
    • add_repositories: add repository url
    • add_packages: add package dependencies to target
    • package: package definition
      • set_urls
      • add_versions
      • set_homepage
      • set_description
      • add_deps
      • [on|before|after]_install: custom install script
      • [on|before|after]_test: custom test script
@uael

This comment has been minimized.

Copy link
Author

uael commented Apr 10, 2017

References :

There is many other but they are good one

@waruqi

This comment has been minimized.

Copy link
Collaborator

waruqi commented Apr 10, 2017

Great! I have finished the xmake repo plugin task to manage repositories.

for example:

$ xmake repo --add origin https://github.com/tboox/xmake-repo.git
$ xmake repo --add other-repo https://github.com/xxxx/other-repo.git

Or use global add_repositories api in xmake.lua:

add_repositories("origin https://github.com/tboox/xmake-repo.git")
add_repositories("other-repo https://github.com/other/other-repo.git")

And the xmake-repo is the default repository, we need not add it manually.

@waruqi

This comment has been minimized.

Copy link
Collaborator

waruqi commented Apr 10, 2017

And we can add requires and add package dependencies (developing ..)

Add requires:

add_requires("tboox.tbox >=1.5.1", "zlib >=1.2.11")
add_requires("zlib master")
add_requires("xmake-repo@tboox.tbox >=1.5.1")
add_requires("https://github.com/tboox/tbox.git@tboox.tbox >=1.5.1")

Add package dependencies:

target("test")
    add_packages("tboox.tbox", "zlib")

If you run xmake require or xmake require --install, xmake will install and update all required packages to local project repository directory.

@uael

This comment has been minimized.

Copy link
Author

uael commented Apr 10, 2017

It's all great 👍
Actually this repo implementation is really nice and consistent. Even if I'm afraid about the future size of the xmake-repo.
A remaining task will be to automate packaging and publication on user repos.
I'm looking for semantic versioner, this is the yarn/npm one, and lua implementations.

@uael

This comment has been minimized.

Copy link
Author

uael commented Apr 10, 2017

How did you plan to handle that case?

add_requires("https://github.com/tboox/tbox.git@tboox.tbox >=1.5.1")
@TitanSnow

This comment has been minimized.

Copy link
Contributor

TitanSnow commented Apr 10, 2017

Excuse me, I'm wondering what's the advantage of xmake repo that git submodule doesn't have.

@waruqi

This comment has been minimized.

Copy link
Collaborator

waruqi commented Apr 11, 2017

@uael I just place xmake.lua and other some text files (.e.g *.patch) in xmake-repo and I did not intend to place the binary file in this repository.
And you can create yourself private repository to place some packages and binary files for each archs and platforms.So it will not be too large.

  • add_requires("https://github.com/tboox/tbox.git@tboox.tbox >=1.5.1")

add_requires can process it directly If this project has complete tags like 1.2.1 or v1.x.x and it will define a package named tboox.tbox. We can use add_packages("tboox.tbox") to add package denpence to target.

If this git project has not tags, we can set a branch name, for example: add_requires("https://github.com/tboox/tbox.git@tboox.tbox master")

Or you can directly define package("tboox.tbox") to describe custem rules in xmake.lua if cannot get git tags, for example:

package("tboox.tbox")
      add_versions("1.5.1", "1.6.0", "1.6.1") -- we can add a version list if no git tags
      set_url("https://github.com/tboox/tbox/archive/v$(v1).$(v2).$(v3).tar.gz")
      on_build(function ()
           -- if need
      end)
      on_package(function ()
           -- if need
      end)

add_requires("tboox.tbox >=1.5.1")

We also use package("xxx") to describe every package in xmake-repo or other private repositories.

@waruqi

This comment has been minimized.

Copy link
Collaborator

waruqi commented Apr 11, 2017

@TitanSnow git submodule is great, but xmake repo and xmake require will provide more convenient cross-platform package management.

Some advantages:

  • Do not need user to manage each third-party package
  • Better version control
  • Support .tar.gz or .zip package if the dependent package does not have a git repo, but git submodule need it.
  • Automated package compilation and linking to the given target.
  • Support multiple platforms, architectures and languanges.

In most cases, you only need to add two lines in xmake.lua to link dependent package automatically which exists in xmake-repo.

add_requires("pcre2 >=1.2.0", "zlib >= 1.2.11")
target("test")
    add_packages("pcre2", "zlib")

Then run:

$ xmake 

And it will show tips when the version of package dependences have been changed in xmake.lua

This project exists outdated package dependences, update them now? y (y / n)
y
installing and updating outdated package dependences ... (call `xmake require` automatically)
building project .. (`xmake build`)
@waruqi

This comment has been minimized.

Copy link
Collaborator

waruqi commented Apr 11, 2017

@uael semver.lua is great 👍. I will consider using it.

@TitanSnow

This comment has been minimized.

Copy link
Contributor

TitanSnow commented Apr 11, 2017

Sounds like fun. Like it❤️. May it be hard work?

@uael

This comment has been minimized.

Copy link
Author

uael commented Apr 11, 2017

@TitanSnow actually the following isn't possible with git submodule:
foo depends on bar;
foo depends on baz;
bar depends on baz;
then with package management the foo vendor dir look like:

vendor
|__ bar
   |__ ..
   :
|__ baz
   |__ ..
   :
@uael

This comment has been minimized.

Copy link
Author

uael commented Apr 11, 2017

In the case of provided binary files i just dunno how to handle it properly, wondering what's your feeling about it

@TitanSnow

This comment has been minimized.

Copy link
Contributor

TitanSnow commented Apr 11, 2017

I got it @uael

@TitanSnow

This comment has been minimized.

Copy link
Contributor

TitanSnow commented Apr 11, 2017

As for me I prefer "code once, compile everywhere" @uael

@uael

This comment has been minimized.

Copy link
Author

uael commented Apr 11, 2017

"code once, compile everywhere" does not prevent to avoid compile time on large build with a bunch of dependencies 👍

@uael

This comment has been minimized.

Copy link
Author

uael commented Apr 11, 2017

@waruqi repositories shouldn't being cloned globally ? Like on on ~/.cache/xmake/repositories/

uael@leopard:~/.cache$ tree xmake/
xmake/
└── repositories
    └── xmake-repo
        ├── libuv # vendor level
        │   ├── libuv # package level
        │   │   ├── 0_1_0.lua # contains specific version informations
        │   │   ├── 0_1_3.lua
        │   │   ├── 0_2_0.lua
        │   │   ├── 0_2_6.lua
        │   │   └── xmake.lua  # general package informations
        │   └── xmake.lua # general vendor informations
        ├── tboox
        │   ├── gbox
        │   │   ├── 1_0_1.lua
        │   │   ├── 1_0_2.lua
        │   │   ├── 1_0_3.lua
        │   │   └── xmake.lua
        │   ├── tbox
        │   │   ├── 1_5_3.lua
        │   │   ├── 1_6_0.lua
        │   │   ├── 1_6_1.lua
        │   │   └── xmake.lua
        │   └── xmake.lua
        └── zlib
            ├── xmake.lua
            └── zlib
                ├── 1_2_11.lua
                └── xmake.lua

9 directories, 18 files

With this structure we can clone the xmake-repo on xmake install and cache packages and versions until the next repository pull. When another repository is added, the cache is updated.
Then global package are downloaded in ~/.cache/xmake/packages/, and local packages in $(buildir)/packages/

@TitanSnow

This comment has been minimized.

Copy link
Contributor

TitanSnow commented Apr 11, 2017

Seems right

@TitanSnow

This comment has been minimized.

Copy link
Contributor

TitanSnow commented Apr 11, 2017

Maybe git large file storage is suitable for you @uael

@waruqi

This comment has been minimized.

Copy link
Collaborator

waruqi commented Apr 11, 2017

@uael Yes, I'm planning to clone repositories globally in the first time and then only need to git pull to update them when the repositories need be updated. And I will continue to optimize it in future.

@waruqi

This comment has been minimized.

Copy link
Collaborator

waruqi commented Apr 13, 2017

I added a optional mode for add_requires("zlib >=1.2.11 optional") api. If this package is optional dependent, we will do not download and install it automatically by default, unless the user runs xmake require zlib to install it.

For example (projectdir/xmake.lua):

-- define package
package("mbedtls")
    set_url("git@github.com:ARMmbed/mbedtls.git")
    on_build(function (package)
    end)
    on_install(function (package)
    end)
package_end()

-- requires
add_requires("mbedtls master optional")    -- optional dependent package
add_requires("zlib >=1.2.11", "xmake-repo@tboox.tbox >=1.5.1")
add_requires("git@github.com:glennrp/libpng.git@libpng >=1.6.28")
@uael

This comment has been minimized.

Copy link
Author

uael commented Apr 13, 2017

Love it 👍
Can be really useful for multiple and optional backend for example, is there a way to test if installed with the lua api ?

@waruqi

This comment has been minimized.

Copy link
Collaborator

waruqi commented Apr 13, 2017

You need test installed packages? xmake will first detect whether a package has been installe, then install it if it does not exist.

@uael

This comment has been minimized.

Copy link
Author

uael commented Apr 14, 2017

More I look at your commits on repo, more I seriously envisage to use lua 👍
Forgot to answer you! No I mean add some defines, for example, to know in the code if an optional package is installed or not.

@waruqi

This comment has been minimized.

Copy link
Collaborator

waruqi commented Apr 14, 2017

you can use is_package("xxx"), for example:

-- require package for `xmake config --zlib=y|n` or checking and installing automatically
add_requires("zlib >1.2")

target("xxxx")
    set_kind("binary")
    add_files("*.c")
    
    if is_package("zlib") then
         add_defines("-DZLIB_ENABLED")
    end

is_package will return true If the zlib package is enabled and installed.

@uael

This comment has been minimized.

Copy link
Author

uael commented Apr 19, 2017

#75

@waruqi

This comment has been minimized.

Copy link
Collaborator

waruqi commented Apr 20, 2017

I rename xmake deps action to xmake require. for example

# install all package dependencies in xmake.lua
$ xmake [q|require]

# install the given required packages manually
$ xmake require zlib tboox.tbox
$ xmake require "zlib >=1.2.11" "tboox.tbox master"
$ xmake require "git@github.com:tboox/tbox.git@tboox.tbox >=1.6.0 <1.6.1 || master"
@TitanSnow

This comment has been minimized.

Copy link
Contributor

TitanSnow commented Apr 20, 2017

By the way, may I ask how soon repo would be merged back to master? Or never? @waruqi

@waruqi

This comment has been minimized.

Copy link
Collaborator

waruqi commented May 24, 2017

@uael I'm improving the build script on windows and installation, linking, and cross-compilation are not yet implemented. But you can test it first. for example:

$ cd tests/projects/deps_console_c
$ xmake require 
$ xmake require -v -f
...
@uael

This comment has been minimized.

Copy link
Author

uael commented May 24, 2017

@waruqi The command took really long time, did you know why ?

@waruqi

This comment has been minimized.

Copy link
Collaborator

waruqi commented May 24, 2017

@uael You can add -v to see the verbose output. Maybe clone repositories, test host speed, get git tags, download and build package...
I will optimize performance (cache, delay handing ..) in the future, but it's not the point now.

@uael

This comment has been minimized.

Copy link
Author

uael commented May 24, 2017

It freeze here updating repositories .. and after

pulling repository(xmake-repo): https://github.com/tboox/xmake-repo.git to /home/uael/.xmake/repositories/xmake-repo ..
git     pull origin master
From https://github.com/tboox/xmake-repo
 * branch            master     -> FETCH_HEAD
Already up-to-date.
ok

for 30s 1m without verbose then it fail on install with:

  => building tboox.tbox-v1.6.0 .. 
xmake f -p linux -a x86_64 -c
checking for the c include unistd.h ... no
checking for the c include sys/socket.h ... no
checking for the c include locale.h ... no
checking for the c include sys/poll.h ... no
checking for the c include signal.h ... no
checking for the c include sys/uio.h ... no
checking for the c include math.h ... no
checking for the c include wchar.h ... no
checking for the c include unistd.h ... no
checking for the c include unistd.h ... no
checking for the c include ifaddrs.h ... no
checking for the links polarssl ... no
checking for the c include string.h ... no
checking for the c include wchar.h ... no
checking for the c include math.h ... no
checking for the c include string.h ... no
checking for the c include math.h ... no
checking for the c include wchar.h ... no
checking for the c include math.h ... no
checking for the c include string.h ... no
checking for the c include stdlib.h ... no
checking for the c include string.h ... no
checking for the c include wchar.h ... no
checking for the c include sys/sem.h ... no
checking for the c include sys/sendfile.h ... no
checking for the c include string.h ... no
checking for the c include string.h ... no
checking for the c include string.h ... no
checking for the links sqlite3 ... no
checking for the c include sys/uio.h ... no
checking for the c include wchar.h ... no
checking for the c include sys/wait.h ... no
checking for the c include math.h ... no
checking for the c include string.h ... no
checking for the c include time.h ... no
checking for the c include unistd.h ... no
checking for the c include unistd.h ... no
checking for the c include math.h ... no
checking for the c include string.h ... no
checking for the c include math.h ... no
checking for the c include regex.h ... no
checking for the c include math.h ... no
checking for the c include wchar.h ... no
checking for the c include sys/uio.h ... no
checking for the c include math.h ... no
checking for the c type wchar_t ... no
checking for the c include pthread.h ... no
checking for the c include signal.h ... no
checking for the c include string.h ... no
checking for the c include spawn.h ... no
checking for the c include time.h ... no
checking for the c include math.h ... no
checking for the c include execinfo.h ... no
checking for the c include sched.h ... no
checking for the c include math.h ... no
checking for the c include wchar.h ... no
checking for the c include unistd.h ... no
checking for the c include dlfcn.h ... no
checking for the c include regex.h ... no
checking for the c include unistd.h ... no
checking for the c include wchar.h ... no
checking for the c include sys/uio.h ... no
checking for the c include sys/epoll.h ... no
checking for the c include sys/sem.h ... no
checking for the c include pthread.h ... no
checking for the links ssl, crypto ... no
checking for the c include sys/stat.h ... no
checking for the c include pthread.h ... no
checking for the links z ... no
checking for the c include time.h ... no
checking for the c include math.h ... no
checking for the c include signal.h ... no
checking for the c include math.h ... no
checking for the c include string.h ... no
checking for the c include string.h ... no
checking for the c include string.h ... no
checking for the c include pthread.h ... no
checking for the c include string.h ... no
checking for the c include stdio.h ... no
checking for the c include unistd.h ... no
checking for the c include wchar.h ... no
checking for the c include string.h ... no
checking for the c include math.h ... no
checking for the c include semaphore.h ... no
checking for the c include math.h ... no
checking for the c include math.h ... no
checking for the c include unistd.h ... no
checking for the links pcre ... no
checking for the c include sys/time.h ... no
checking for the c include pthread.h ... no
checking for the c include wchar.h ... no
checking for the links pcre2-8 ... no
checking for the c include wchar.h ... no
checking for the links mysqlclient ... no
checking for the c include dirent.h ... no
checking for the c include math.h ... no
checking for the c include string.h ... no
checking for the c include math.h ... no
checking for the c include math.h ... no
checking for the c include wchar.h ... no
checking for the c include sys/epoll.h ... no
checking for the c include string.h ... no
checking for the c include wchar.h ... no
checking for the c include wchar.h ... no
checking for the c include math.h ... no
checking for the links mbedtls, mbedcrypto, mbedx509 ... no
checking for the c include pthread.h ... no
checking for the c include wchar.h ... no
checking for the c include math.h ... no
checking for the c include math.h ... no
checking for the c include signal.h ... no
checking for the c include math.h ... no
checking for the c include unistd.h ... no
checking for the c include math.h ... no
checking for the c include math.h ... no
checking for the c include math.h ... no
checking for the c include sys/resource.h ... no
checking for the c include stdlib.h ... no
checking for the c include unistd.h ... no
checking for the links pthread, dl, m, c ... no
checking for the c include string.h ... no
configure
{
    deprecated = true
,   clean = true
,   polarssl = false
,   object = true
,   arch = x86_64
,   sqlite3 = false
,   info = true
,   smallest = false
,   cc = gcc
,   plat = linux
,   wchar = false
,   float = true
,   host = linux
,   charset = true
,   xml = true
,   exception = false
,   pcre2 = false
,   openssl = false
,   zlib = false
,   database = true
,   regex = true
,   ccache = true
,   buildir = build
,   mysql = false
,   mode = release
,   hash = true
,   mbedtls = false
,   coroutine = true
,   zip = true
,   demo = true
,   base = false
,   kind = static
,   pcre = false
}
xmake -r
error: cannot get tool for dc-ar
/usr/local/share/xmake/core/base/os.lua:592: exec(xmake -r) failed(255)!
stack traceback:
    [C]: in function 'error'
    [/usr/local/share/xmake/core/base/os.lua:592]: in function 'raise'
    [/usr/local/share/xmake/core/sandbox/modules/os.lua:318]:
    [/usr/local/share/xmake/core/sandbox/modules/os.lua:262]: in function 'vrun'
    [/usr/local/share/xmake/actions/require/action/build.lua:43]:
    [/usr/local/share/xmake/actions/require/action/build.lua:100]:
@uael

This comment has been minimized.

Copy link
Author

uael commented May 24, 2017

There is a lot more verbose while building i just pasted the end

@waruqi

This comment has been minimized.

Copy link
Collaborator

waruqi commented May 24, 2017

@uael Ok, I will look at it.

@waruqi

This comment has been minimized.

Copy link
Collaborator

waruqi commented May 24, 2017

@uael You can try running xmake require -v -f to force to download package again and give me verbose info.

About this problem, you can create a new issue. 😸

@uael

This comment has been minimized.

Copy link
Author

uael commented May 24, 2017

I will 👍

@waruqi

This comment has been minimized.

Copy link
Collaborator

waruqi commented May 25, 2017

@uael I improved the speed of pulling git refs (multi-jobs) and cache it, you can try it again. But it is still slower in the first time. I will continue to improve it in the future.

@uael

This comment has been minimized.

Copy link
Author

uael commented May 25, 2017

I just updated from the latest repo

uael@aero:~/xmake/tests/projects/deps_console_c$ xmake require -v -f
error: /home/uael/xmake/tests/projects/deps_console_c/xmake.lua:2: attempt to call global 'package' (a nil value)
stack traceback:
    [/home/uael/xmake/tests/projects/deps_console_c/xmake.lua:2]: in main chunk

It's maybe badly installed from me 😸
I run make build then the get.sh with __install_only__ and __local__

@waruqi

This comment has been minimized.

Copy link
Collaborator

waruqi commented May 25, 2017

@uael This should be a installation problem. It will be installed to /usr/local directory by default if we run make build; sudo make install. And it will be installed to ~/.local directory if we run scripts/get.sh __local__.

So maybe the previous version has not been uninstalled or updated and you're running the old version of xmake.

You can attempt to run sudo make uninstall to remove old version first and try installing it again.

@waruqi waruqi modified the milestones: package manager, v2.2.1 Jun 3, 2017

@uael

This comment has been minimized.

Copy link
Author

uael commented Jun 7, 2017

Hi @waruqi, how is this going ? I'm currently passing some exams cannot follow as before 😢

@waruqi

This comment has been minimized.

Copy link
Collaborator

waruqi commented Jun 7, 2017

@uael Thank you very much for your contribution and help. The semantic version module is very great now.

I'm implementing find_package to detect and link the dependent packages and I'll finish the rest of the job as soon as possible.

And I haven't had much time to focus on semver module lately. If there is a problem with the semantic version in the future, I will try to modify it and submit pr and issues to you.

Thanks!

@waruqi waruqi referenced this issue Jun 13, 2017

Closed

`find_library` module #122

11 of 11 tasks complete
@waruqi

This comment has been minimized.

Copy link
Collaborator

waruqi commented Aug 14, 2017

@uael I'm so sorry I spent too much time in order to implement the detect modules, so the development of package management was delayed.

Now I'm going to continue to start this job and hope to be able to complete it in v2.2.1 version.

I have merged all patches of the repo branch to dev branch and will remove repo branch.

Because there are too many conflicts, I can only merge code manually, and I'm sorry that I didn't keep the information of contributors. 😭

@waruqi

This comment has been minimized.

Copy link
Collaborator

waruqi commented Sep 16, 2017

A test example:

$ cd xmake/tests/project/requires
$ xmake

I am improving the dependent environment of requires on each platform (git, curl, ..).

@yangjz1125

This comment has been minimized.

Copy link
Contributor

yangjz1125 commented Aug 29, 2018

Can I usr remote package management now?

@waruqi

This comment has been minimized.

Copy link
Collaborator

waruqi commented Aug 29, 2018

@yangjz1125 Not yet, I am processing other issues now.
This feature is also in development, the next version can be used. Please wait some time.

@yangjz1125

This comment has been minimized.

Copy link
Contributor

yangjz1125 commented Aug 29, 2018

Okay, thanks.

@waruqi

This comment has been minimized.

Copy link
Collaborator

waruqi commented Sep 20, 2018

An example: https://github.com/tboox/benchbox

Run:

$ git clone https://github.com/tboox/benchbox.git
$ cd benchbox
$ xmake

package

@waruqi

This comment has been minimized.

Copy link
Collaborator

waruqi commented Oct 17, 2018

@yangjz1125 @uael This feature can be used for v2.2.2

An example project: https://github.com/tboox/benchbox

Documents:

add_requires("tbox 1.6.*", "libpng ~1.16", "zlib")

target("test")
    set_kind("binary")
    add_files("src/*.c") 
    add_packages("tbox", "libpng", "zlib")

@uael

This comment has been minimized.

Copy link
Author

uael commented Oct 17, 2018

Very good work here, repo/package are a well designed approach. I really have to pick a side project to try it out!

@uael

This comment has been minimized.

Copy link
Author

uael commented Oct 17, 2018

I just finished the docs, thank you for quoting me 😃 Very accurate and understandable by the way

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment