Remote package management #69

uael · 2017-04-10T12:57:03Z

A package manager or package management system is a collection of software tools that automates the process of installing, upgrading, configuring, and removing computer programs for a computer's operating system in a consistent manner.

A package manager deals with packages, distributions of software and data in archive files. Packages contain metadata, such as the software's name, description of its purpose, version number, vendor, checksum, and a list of dependencies necessary for the software to run properly. Upon installation, metadata is stored in a local package database. Package managers typically maintain a database of software dependencies and version information to prevent software mismatches and missing prerequisites. They work closely with software repositories, binary repository managers, and app stores.

Source: package manager

Remote package management roadmap:

The text was updated successfully, but these errors were encountered:

uael · 2017-04-10T13:01:41Z

References :

There is many other but they are good one

waruqi · 2017-04-10T13:08:34Z

Great! I have finished the xmake repo plugin task to manage repositories.

for example:

$ xmake repo --add origin https://github.com/tboox/xmake-repo.git
$ xmake repo --add other-repo https://github.com/xxxx/other-repo.git

Or use global add_repositories api in xmake.lua:

add_repositories("origin https://github.com/tboox/xmake-repo.git")
add_repositories("other-repo https://github.com/other/other-repo.git")

And the xmake-repo is the default repository, we need not add it manually.

waruqi · 2017-04-10T13:16:53Z

And we can add requires and add package dependencies (developing ..)

Add requires:

add_requires("tboox.tbox >=1.5.1", "zlib >=1.2.11")
add_requires("zlib master")
add_requires("xmake-repo@tboox.tbox >=1.5.1")
add_requires("https://github.com/tboox/tbox.git@tboox.tbox >=1.5.1")

Add package dependencies:

target("test")
    add_packages("tboox.tbox", "zlib")

If you run xmake require or xmake require --install, xmake will install and update all required packages to local project repository directory.

uael · 2017-04-10T15:42:01Z

It's all great 👍
Actually this repo implementation is really nice and consistent. Even if I'm afraid about the future size of the xmake-repo.
A remaining task will be to automate packaging and publication on user repos.
I'm looking for semantic versioner, this is the yarn/npm one, and lua implementations.

uael · 2017-04-10T20:45:17Z

How did you plan to handle that case?

add_requires("https://github.com/tboox/tbox.git@tboox.tbox >=1.5.1")

ghost · 2017-04-10T23:17:50Z

Excuse me, I'm wondering what's the advantage of xmake repo that git submodule doesn't have.

waruqi · 2017-04-11T02:39:24Z

@uael I just place xmake.lua and other some text files (.e.g *.patch) in xmake-repo and I did not intend to place the binary file in this repository.
And you can create yourself private repository to place some packages and binary files for each archs and platforms.So it will not be too large.

add_requires("https://github.com/tboox/tbox.git@tboox.tbox >=1.5.1")

add_requires can process it directly If this project has complete tags like 1.2.1 or v1.x.x and it will define a package named tboox.tbox. We can use add_packages("tboox.tbox") to add package denpence to target.

If this git project has not tags, we can set a branch name, for example: add_requires("https://github.com/tboox/tbox.git@tboox.tbox master")

Or you can directly define package("tboox.tbox") to describe custem rules in xmake.lua if cannot get git tags, for example:

package("tboox.tbox")
      add_versions("1.5.1", "1.6.0", "1.6.1") -- we can add a version list if no git tags
      set_url("https://github.com/tboox/tbox/archive/v$(v1).$(v2).$(v3).tar.gz")
      on_build(function ()
           -- if need
      end)
      on_package(function ()
           -- if need
      end)

add_requires("tboox.tbox >=1.5.1")

We also use package("xxx") to describe every package in xmake-repo or other private repositories.

waruqi · 2017-04-11T02:53:00Z

@titansnow git submodule is great, but xmake repo and xmake require will provide more convenient cross-platform package management.

Some advantages:

Do not need user to manage each third-party package
Better version control
Support .tar.gz or .zip package if the dependent package does not have a git repo, but git submodule need it.
Automated package compilation and linking to the given target.
Support multiple platforms, architectures and languanges.

In most cases, you only need to add two lines in xmake.lua to link dependent package automatically which exists in xmake-repo.

add_requires("pcre2 >=1.2.0", "zlib >= 1.2.11")
target("test")
    add_packages("pcre2", "zlib")

Then run:

$ xmake

And it will show tips when the version of package dependences have been changed in xmake.lua

This project exists outdated package dependences, update them now? y (y / n)
y
installing and updating outdated package dependences ... (call `xmake require` automatically)
building project .. (`xmake build`)

waruqi · 2017-04-11T03:10:50Z

@uael semver.lua is great 👍. I will consider using it.

ghost · 2017-04-11T05:29:34Z

Sounds like fun. Like it:heart:. May it be hard work?

uael · 2017-04-11T05:44:10Z

@titansnow actually the following isn't possible with git submodule:
foo depends on bar;
foo depends on baz;
bar depends on baz;
then with package management the foo vendor dir look like:

vendor
|__ bar
   |__ ..
   :
|__ baz
   |__ ..
   :

uael · 2017-04-11T05:51:37Z

In the case of provided binary files i just dunno how to handle it properly, wondering what's your feeling about it

ghost · 2017-04-11T05:57:05Z

I got it @uael

ghost · 2017-04-11T06:00:17Z

As for me I prefer "code once, compile everywhere" @uael

uael · 2017-04-11T06:22:58Z

"code once, compile everywhere" does not prevent to avoid compile time on large build with a bunch of dependencies 👍

uael · 2017-04-11T06:27:29Z

@waruqi repositories shouldn't being cloned globally ? Like on on ~/.cache/xmake/repositories/

uael@leopard:~/.cache$ tree xmake/
xmake/
└── repositories
    └── xmake-repo
        ├── libuv # vendor level
        │   ├── libuv # package level
        │   │   ├── 0_1_0.lua # contains specific version informations
        │   │   ├── 0_1_3.lua
        │   │   ├── 0_2_0.lua
        │   │   ├── 0_2_6.lua
        │   │   └── xmake.lua  # general package informations
        │   └── xmake.lua # general vendor informations
        ├── tboox
        │   ├── gbox
        │   │   ├── 1_0_1.lua
        │   │   ├── 1_0_2.lua
        │   │   ├── 1_0_3.lua
        │   │   └── xmake.lua
        │   ├── tbox
        │   │   ├── 1_5_3.lua
        │   │   ├── 1_6_0.lua
        │   │   ├── 1_6_1.lua
        │   │   └── xmake.lua
        │   └── xmake.lua
        └── zlib
            ├── xmake.lua
            └── zlib
                ├── 1_2_11.lua
                └── xmake.lua

9 directories, 18 files

With this structure we can clone the xmake-repo on xmake install and cache packages and versions until the next repository pull. When another repository is added, the cache is updated.
Then global package are downloaded in ~/.cache/xmake/packages/, and local packages in $(buildir)/packages/

ghost · 2017-04-11T06:28:50Z

Seems right

ghost · 2017-04-11T06:30:17Z

Maybe git large file storage is suitable for you @uael

waruqi · 2017-04-11T06:57:21Z

@uael Yes, I'm planning to clone repositories globally in the first time and then only need to git pull to update them when the repositories need be updated. And I will continue to optimize it in future.

waruqi · 2017-04-13T02:52:05Z

I added a optional mode for add_requires("zlib >=1.2.11 optional") api. If this package is optional dependent, we will do not download and install it automatically by default, unless the user runs xmake require zlib to install it.

For example (projectdir/xmake.lua):

-- define package
package("mbedtls")
    set_url("git@github.com:ARMmbed/mbedtls.git")
    on_build(function (package)
    end)
    on_install(function (package)
    end)
package_end()

-- requires
add_requires("mbedtls master optional")    -- optional dependent package
add_requires("zlib >=1.2.11", "xmake-repo@tboox.tbox >=1.5.1")
add_requires("git@github.com:glennrp/libpng.git@libpng >=1.6.28")

uael · 2017-04-13T09:46:16Z

Love it 👍
Can be really useful for multiple and optional backend for example, is there a way to test if installed with the lua api ?

waruqi · 2017-04-13T10:28:27Z

You need test installed packages? xmake will first detect whether a package has been installe, then install it if it does not exist.

uael · 2017-04-14T11:22:22Z

More I look at your commits on repo, more I seriously envisage to use lua 👍
Forgot to answer you! No I mean add some defines, for example, to know in the code if an optional package is installed or not.

waruqi · 2017-04-14T13:19:40Z

you can use is_package("xxx"), for example:

-- require package for `xmake config --zlib=y|n` or checking and installing automatically
add_requires("zlib >1.2")

target("xxxx")
    set_kind("binary")
    add_files("*.c")
    
    if is_package("zlib") then
         add_defines("-DZLIB_ENABLED")
    end

is_package will return true If the zlib package is enabled and installed.

uael · 2017-04-19T22:37:25Z

#75

waruqi · 2017-04-20T02:32:21Z

I rename xmake deps action to xmake require. for example

# install all package dependencies in xmake.lua
$ xmake [q|require]

# install the given required packages manually
$ xmake require zlib tboox.tbox
$ xmake require "zlib >=1.2.11" "tboox.tbox master"
$ xmake require "git@github.com:tboox/tbox.git@tboox.tbox >=1.6.0 <1.6.1 || master"

ghost · 2017-04-20T02:52:40Z

By the way, may I ask how soon repo would be merged back to master? Or never? @waruqi

uael · 2017-05-25T13:29:39Z

I just updated from the latest repo

uael@aero:~/xmake/tests/projects/deps_console_c$ xmake require -v -f
error: /home/uael/xmake/tests/projects/deps_console_c/xmake.lua:2: attempt to call global 'package' (a nil value)
stack traceback:
    [/home/uael/xmake/tests/projects/deps_console_c/xmake.lua:2]: in main chunk

It's maybe badly installed from me 😸
I run make build then the get.sh with __install_only__ and __local__

waruqi · 2017-05-25T13:48:35Z

@uael This should be a installation problem. It will be installed to /usr/local directory by default if we run make build; sudo make install. And it will be installed to ~/.local directory if we run scripts/get.sh __local__.

So maybe the previous version has not been uninstalled or updated and you're running the old version of xmake.

You can attempt to run sudo make uninstall to remove old version first and try installing it again.

uael · 2017-06-07T07:22:49Z

Hi @waruqi, how is this going ? I'm currently passing some exams cannot follow as before 😢

waruqi · 2017-06-07T09:00:10Z

@uael Thank you very much for your contribution and help. The semantic version module is very great now.

I'm implementing find_package to detect and link the dependent packages and I'll finish the rest of the job as soon as possible.

And I haven't had much time to focus on semver module lately. If there is a problem with the semantic version in the future, I will try to modify it and submit pr and issues to you.

Thanks!

waruqi · 2017-08-14T03:30:48Z

@uael I'm so sorry I spent too much time in order to implement the detect modules, so the development of package management was delayed.

Now I'm going to continue to start this job and hope to be able to complete it in v2.2.1 version.

I have merged all patches of the repo branch to dev branch and will remove repo branch.

Because there are too many conflicts, I can only merge code manually, and I'm sorry that I didn't keep the information of contributors. 😭

waruqi · 2017-09-16T05:26:55Z

A test example:

$ cd xmake/tests/project/requires
$ xmake

I am improving the dependent environment of requires on each platform (git, curl, ..).

ColorfulOrangeCN · 2018-08-29T01:46:34Z

Can I usr remote package management now?

waruqi · 2018-08-29T02:00:19Z

@yangjz1125 Not yet, I am processing other issues now.
This feature is also in development, the next version can be used. Please wait some time.

ColorfulOrangeCN · 2018-08-29T02:38:55Z

Okay, thanks.

waruqi · 2018-09-20T07:00:35Z

An example: https://github.com/tboox/benchbox

Run:

$ git clone https://github.com/tboox/benchbox.git
$ cd benchbox
$ xmake

waruqi · 2018-10-17T13:41:03Z

@yangjz1125 @uael This feature can be used for v2.2.2

An example project: https://github.com/tboox/benchbox

Documents:

add_requires("tbox 1.6.*", "libpng ~1.16", "zlib")

target("test")
    set_kind("binary")
    add_files("src/*.c") 
    add_packages("tbox", "libpng", "zlib")

uael · 2018-10-17T19:13:19Z

Very good work here, repo/package are a well designed approach. I really have to pick a side project to try it out!

uael · 2018-10-17T19:41:38Z

I just finished the docs, thank you for quoting me 😃 Very accurate and understandable by the way

waruqi · 2019-05-30T12:48:03Z

I have almost completed this feature, and I will continue to improve it and the package repository. xmake-repo

Documents: https://xmake.io/#/home?id=remote-dependency-mode

waruqi added discussion feature request in progress labels Apr 10, 2017

waruqi modified the milestones: package manager, v2.2.1 Jun 3, 2017

waruqi mentioned this issue Jun 13, 2017

find_library module #122

Closed

11 tasks

waruqi mentioned this issue Aug 29, 2017

switch build system dosemu2/dosemu2#393

Open

waruqi pinned this issue Jan 7, 2019

waruqi mentioned this issue Jan 29, 2019

Improve add_requires and find_package to integrate the 3rd package manager #339

Closed

31 tasks

waruqi mentioned this issue Mar 25, 2019

Support to apply patches for packages in repository #374

Closed

3 tasks

waruqi closed this as completed May 30, 2019

waruqi modified the milestones: v2.3.1, v2.2.7, v2.2.6 May 30, 2019

waruqi added done and removed in progress labels May 30, 2019

waruqi unpinned this issue Nov 20, 2019

Remote package management #69

Remote package management #69

Comments

uael commented Apr 10, 2017 • edited by waruqi Loading

uael commented Apr 10, 2017 • edited by waruqi Loading

waruqi commented Apr 10, 2017

waruqi commented Apr 10, 2017 • edited Loading

uael commented Apr 10, 2017 • edited Loading

uael commented Apr 10, 2017

ghost commented Apr 10, 2017

waruqi commented Apr 11, 2017 • edited Loading

waruqi commented Apr 11, 2017 • edited Loading

waruqi commented Apr 11, 2017

ghost commented Apr 11, 2017

uael commented Apr 11, 2017 • edited Loading

uael commented Apr 11, 2017 • edited Loading

ghost commented Apr 11, 2017

ghost commented Apr 11, 2017

uael commented Apr 11, 2017

uael commented Apr 11, 2017 • edited Loading

ghost commented Apr 11, 2017

ghost commented Apr 11, 2017 • edited by ghost Loading

waruqi commented Apr 11, 2017 • edited Loading

waruqi commented Apr 13, 2017 • edited Loading

uael commented Apr 13, 2017

waruqi commented Apr 13, 2017

uael commented Apr 14, 2017

waruqi commented Apr 14, 2017 • edited Loading

uael commented Apr 19, 2017

waruqi commented Apr 20, 2017 • edited Loading

ghost commented Apr 20, 2017

uael commented May 25, 2017

waruqi commented May 25, 2017

uael commented Jun 7, 2017

waruqi commented Jun 7, 2017

waruqi commented Aug 14, 2017 • edited Loading

waruqi commented Sep 16, 2017

ColorfulOrangeCN commented Aug 29, 2018

waruqi commented Aug 29, 2018 • edited Loading

ColorfulOrangeCN commented Aug 29, 2018

waruqi commented Sep 20, 2018

waruqi commented Oct 17, 2018

uael commented Oct 17, 2018

uael commented Oct 17, 2018

waruqi commented May 30, 2019

uael commented Apr 10, 2017 •

edited by waruqi

Loading

uael commented Apr 10, 2017 •

edited by waruqi

Loading

waruqi commented Apr 10, 2017 •

edited

Loading

uael commented Apr 10, 2017 •

edited

Loading

waruqi commented Apr 11, 2017 •

edited

Loading

waruqi commented Apr 11, 2017 •

edited

Loading

uael commented Apr 11, 2017 •

edited

Loading

uael commented Apr 11, 2017 •

edited

Loading

uael commented Apr 11, 2017 •

edited

Loading

ghost commented Apr 11, 2017 •

edited by ghost

Loading

waruqi commented Apr 11, 2017 •

edited

Loading

waruqi commented Apr 13, 2017 •

edited

Loading

waruqi commented Apr 14, 2017 •

edited

Loading

waruqi commented Apr 20, 2017 •

edited

Loading

waruqi commented Aug 14, 2017 •

edited

Loading

waruqi commented Aug 29, 2018 •

edited

Loading