-
-
Notifications
You must be signed in to change notification settings - Fork 45
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Need some help to package Perl6 modules for Debian.. #117
Comments
Removal of precomp files would be the responsibility of rakudo (i.e. the files should be removed internally just like all the other source/resource/script files), because its the only thing that actually knows where they are or which one is which. This is not something that is implemented yet in When precomp files are created they take the current rakudo version (and other factors) to generate the final file name (some sha1). This means it is not technically necessary to delete old precomp files when you upgrade rakudo because:
|
I understand that running a Perl6 program can generate pre-compiled files on the fly and that's not a problem for people who are trying Perl6. However I believe that on-the-fly precompilation may eventually have undesirables side effects:
I guess that these 2 issues can be avoided by pre-compiling Perl6 modules with zef as installation time. I.e. run I'd rather avoid leaving pre-compilation files around. I grant you that they are harmless, but they may add up to significant space over the years, especially on ARM system. I'm pretty sure that I'll get bug report if a cleanup is not done from time to time. I guess that I could try to remove all pre-compiled files (with Does that sound reasonable ? All the best |
There is no easy solution to this problem. See: python Precompilation will only occur by direct action from rakudo itself, not a package manager like zef. A package manager has absolutely no way of determining what other modules a specific module in a distribution depends on so it has no way of building a DAG to determine the correct order to precompile the files in/against. This is why it must happen on-the-fly, because the only way to resolve this graph is by actually compiling and running the program. When you install a module it does get precompiled, but this is essentially just doing a Zef has no way of cleaning up precompiled files. Ideally rakudo would provide a mechanism such that external tools could determine the precompilation file location of a given name - this would allow zef to add such a feature. But zef cannot just guess or determine the filename, because these can change based on a repo format that is used by rakudo (hence needing a mechanism in rakudo to provide this info). If you really wanted to do this manually you could probably scan the short-id folder under an installation directory, look inside all the files to determine what the sha1 in the directory name maps to name wise, then use that to somehow filter the precomp directory (I'm not sure if this is possible). A more ideal solution would be to create a custom CompUnit::Repository that adds the features you wants. Unfortunately I'm not sure how to handle the precompilation portion of this. What I can provide is an example that does not precompile. This example also does not have an install/uninstall method - but this may be of interest to you because you probably don't want zef uninstalling modules installed by apt-get (but you could add these methods and add bits to install/uninstall that update the appropriate dpkg/apt files to reflect such a change). The repository represents a single distribution anyway (so only a single file, blah.tar.gz), so installation and uninstallation is really just moving/deleting a single file. The example is based on a .tar.gz file, but all its doing is extracting files on the fly to stdout (not to a file) that get loaded so if you can output files from a dpkg to stdout you could do something similar. |
Hi Sorry for the long delay, I got side tracked by new year and incoming Debian release. I may have a workable plan to ensure installation, upgrade and removal of pre-compiled files. This should possible thanks to the information stored in Debian package manager. BTW, many thanks for providing the Here we go: package buildPackage will deliver:
Build will be done only to enable tests. No build artifact will be shipped in the package. package installationFiles will be installed by package manager in Then a post install script is run:
Note file list will look like:
❓ should the lock files be shipped or discarded ? on module deletionpackage scripts will:
on module upgrade:package scripts will:
on rakudo upgraderakudo package scripts will:
❓ should this be triggered when moar or nqp are upgraded as well ? All in all, we should end up with a pretty clean What do you think ? Does that sound reasonable ? Did I miss something ? All the best |
You sound like you have a good idea of what you need to do. I'm not sure how the precomp stuff will work in practice - module upgrading and re installation may be expected to re-precompile against the new distributions (reverse dependencies). If it doesn't work you could probably delete/reinstall all of the dependencies/reverse-dependencies like listed under module upgrade
Give it a shot! 👍 |
Will do. Don't hold your breath :-) BTW, to satisfy Debian policy (and formally allow us to re-distribute your software) could you mention the copyright owners and the license under which this software is released ? (I assume Artistic-2, but assumptions don't work in legal world) All the best |
LICENSE has been added |
Thanks for the LICENSE file. What is the Should this be kept or removed ? All the best |
Version file is generated by Rakudo when installing modules. I'm not sure how you want to handle that part, but the version file contains the "repo-version" CURI can use to read the module from its installed location (it is not zef specific, it is specific to each CompUnit::Repository). This is so when you do a regular rakudo upgrade to a newer version that uses a different install format it can automatically upgrade the current repo to the new format. |
How it works is say you install zef into a really old rakudo. There will be no With this in mind you should be able to determine how you want to use the |
Thanks for the explanation. I'm still not sure what I should do with Before I tackle this, I need to better understand how precompiled files should be installed (I'm not there yet). For instance, the This is not the case:
Why is the file All the best |
For what it's worth, the extract from strace belows shows that a precompiled file is found in
The trace above was produced with:
Is this behavior expected ? All the best |
BTW, I think zef files should be installed in Do you agree ? |
Yeah you would probably want to put zef in vendor. Regarding precomp: see this discussion https://irclog.perlgeek.de/perl6-toolchain/2017-03-18#i_14287090 |
dod38fr: I have spent loads of time thinking about how rakudo's precompilation can work with distribution packaging and have implemented it in a way so all your issues should actually be fixed already. Rakudo precompiles modules at installation time. When I'm done with my work on the toolchain, you won't need zef for packaging modules. Zef is a tool for the end user. It has much more functionality than you'd need for a simple job as packaging a module. This functionality might even be hurtful. When creating a distro package you will want installation to fail loudly if a dependency is missing, while zef will happily grab the dependency for you. Rakudo already contains a minimal module installer meant explicitly for packagers (tools/install-dist.pl). There's still an issue with the generated precomp files which I'm working on and the open question of how to include custom build steps (like compiling the C-library for Inline::Perl5). Once those are solved, packaging modules should be downright trivial. Please talk to me before wasting any more time on solving issues that already have a proper solution. My intention has always been to solve those issues for you. The way I envision packaging is:
Regarding your issue with rakudo not using the precomp files created at installation time, please try again with RAKUDO_MODULE_DEBUG=1 in your environment. The debug output should tell us why it cannot use those files and instead compiles again. |
Thanks for the
The times are identical whereas
I guess that the times are messed up when I copy files during rakudo packaging. I'll check this point. I need to think more about the other points. I'll get back to you, Thanks |
Hello I agree that using Shipping pre-compiled file is indeed simpler than compiling in post-install script. That would be my preferred choice. But I wonder what happens when rakudo is updated ? I guess that for a regular user, existing modules are precompiled when needed and the result is saved in I'm more worried for the case where a Perl6 web server is running. In this case, pre-compiled modules would also be precompiled when needed and the result would be saved for the user that run the server (www-data on Debian). Even if the exploit might not be easy, this raise a security issue where the web server is able to write files that will be executed later. Another problem arise for server running with a low privilege user (like user nobody): rakudo may fail because the low privilege user has no possibility to save the precompiled files. All the best |
Shipping pre-compiled file is indeed simpler than compiling in post-install
script. That would be my preferred choice.
But I wonder what happens when rakudo is updated ?
An upgrade of the rakudo package would have to trigger a re-build of the Perl
6 module packages in the distributor's build infrastructure. Thus an upgrade
of rakudo would mean getting updated module packages for the user. These
module packages contain the appropriate precomp files. This is also the way the
user gets rid of the old, no longer useful precomp files.
AFAIK on e.g. the Open Build Service, simply stating the build time dependency
in the meta data is enough to get an automatic re-build.
|
😮 I did not think about that.. May be I can (ab)use Debian transition mechanism to perform these regular rebuilds. It looks like ocaml and haskell have that kind of setting. I'll ask for advices with my fellow Debian developers. Thanks for the hint :-) |
I think I've found why rakudo does not use the insalled pre-compiled files. As explained before, rakudo recompiles the files because the time stamp of the source is the same as the time stamp of the precompiled file. This is due to a combination of smaller issues (or features):
See for instance the time stamp of these files installed with a Debian package:
This command provides a different result on a rakudo installed with rakudo-star (tested on rakudo-star 2017.01 docker image):
Nevertheless, a 2 second difference might be considered as a close shave... Is it possible to change rakudo so that the time-stamp of the original source file is preserved when creating the source file in All the best |
Yes it's possible. If you look inside an installed META6.json file in
(note it has taken the original meta6.json and turned the previous leaf node, The problem is that you can't force |
I've always thought lying to software is an odd way to reach one's goal.
Well, at least not easily. MoarVM doesn't expose uv_fs_utime, so we'd have to add support to MoarVM, nqp and rakudo first. Though it's good to know, that we at least have the information available. However, I wonder if we shouldn't instead make an effort to get rid of timestamp checking entirely. On file systems like FAT, we are faced with a granularity of 2 whole seconds (!), i.e. half an eternity in computer terms. Issues with time stamps were the reason why I already replaced much of the time stamp checking with sha hashes (e.g. precomp files themselves are hashed). If we record the source file's hash and compare that, we can be absolutely sure instead of having to be cautious. We already only check the dependencies' .modified if the identity (hash) of the whole repo chain changed. For files in an Installation repository, we can even record the hash at installation time and add it to the short-name lookup file. Might even save us a couple of milliseconds when loading a module compared to the timestamp check. |
From what I understand tar file (used inside of Debian package) have a granularity of one second. RPM packages are based on cpio. This program is quite old and I would not be surprised if cpio archives have the same time granularity.
Agreed. Basing build decision of file signature was the cornerstone of old cons build system. This was much more reliable that doing rebuild based on timestamp. Anything that can reduce the number of Perl6 modules rebuilds will be most welcome by Debian project. |
First part just landed: rakudo/rakudo@ca0a743 The only modification time stamp check left is for the top-level module itself in CompUnit::PrecompilationRepository::load. That should be rather easy. |
Btw. this is Inline::Perl5's spec file for creating an RPM package: |
And the second part is done: rakudo/rakudo@ff4a034 We now no longer check any modification time stamps when loading precompilation files. We now fully rely on checksums. This should help with Debian's reproducible builds :) |
Thanks for the updates. I'll check them out once rakudo 2017.05 is out. All the best |
I try to get zef and perl6 modules in Fedora. At the URL https://bugzilla.redhat.com/show_bug.cgi?id=1452985 is the review request for zef. If zef is installed with 'tools/install-dist.pl', it seems not to work. It would be nice if the script 'tools/install-dist.pl' would get some documentation. That it is for installing modules and the example how to use it like it is here in a above comment would be nice. Is it possible to install zef by itself in two steps. First to build it and then to move|copy the files to its destination? |
For me locally
The two step process you ask of works as well:
|
In the first version I did this:
But after removing the BUILD directory the package do not work any more. In the second version I used the script 'tools/install-dist.pl': Using the option '--for=vendor' seems to be needed when installing zef to the buildroot destination specified with the option '--to'. I hope someone will take the review and this will pass it. I will report this here. |
FWIW this is the example for how it's supposed to be used: https://build.opensuse.org/package/view_file/devel:languages:perl6/perl6-Inline-Perl5/perl6-Inline-Perl5.spec?expand=1 |
Any update on this? |
We've been sidetracked by issues we've faced to build rakudo on all architectures supported by Debian. This is now working. We are resuming work on packaging modules for Perl6. @robertlemmen is testing some code to manage pre-compiled files in postinstall scripts. All the best |
@dod38fr thank you for keeping us updated! |
@dod38fr @robertlemmen any news? Also, have you seen this packaging doc on fedora wiki (maybe there's something useful in it)? |
from my point of view: moar, nqp and rakudo seem to be working fine now, zef and tap-harness would be the first to "module" packages that we need. We already have packages for review, but not in the archive yet: The fedora wiki is certainly interesting, I also looked at what arch does for more inspiration. |
Hi The only issue left (which is minor), is the lack of a good old man page for zef. Debian policy requires a man page for each executable installed on a Debian system. I would love to be able to generate a man page from zef documentation. So far the only doc I've seen is the output of Do you have any plan to provide zef usage doc in a structured format that could be transformed in a man page ? All the best |
I've spent a few hours trying to figure out the perl6-zef package on sid. Basically, it doesn't install:
It turns out that while the |
It turns out that while the `rakudo` package does indeed install
`install-dist.p6`, it doesn't install `CompUnit::Repository::Staging` (the
upstream source installs neither, as far as I can see, the installation of
`install-dist.p6` is a simple file copy, specified by `debian/install` in
the rakudo package source)
CompUnit::Repository::Staging gets installed as part of the CORE dist into
inst#/usr/share/perl6/core during `make install`
|
Ah, interesting! It seems that the Debian package : ~/gitwrk/github.com/rakudo/rakudo $ ~/.private-install/opt/rakudo/bin/raku ./tools/install-dist.p6
Type check failed in binding to parameter '$meta-file'; expected IO::Path but got Any (Any)
in sub MAIN at ./tools/install-dist.p6 line 59
in block <unit> at ./tools/install-dist.p6 line 35
: ~/gitwrk/github.com/rakudo/rakudo $ /usr/bin/raku ./tools/install-dist.p6
===SORRY!=== Error while compiling ./tools/install-dist.p6
Could not find CompUnit::Repository::Staging in:
inst#/home/levitte/.raku
inst#/usr/share/perl6/site
inst#/usr/share/perl6/vendor
inst#/usr/share/perl6/core
ap#
nqp#
perl5#
at ./tools/install-dist.p6:35 |
Raku is broken on Debian/sid because raku is looking for modules in This issue is tracked by Debian bug 969578 and rakudo/rakudo#3537 On Debian, you should downgrade to testing version to get a working rakudo package. All the best |
Heh, I scrolled down enough on that report to see the path error just moments ago. Thanks, that explains it. |
Another option is to observe that 2020.09 was release just a couple of hours ago 😉. A quick uscan / uupdate of the moarvm, nqp and rakudo source packages, |
I've used install-dist.raku successfully for gentoo linux. |
Hello After all, I've decided to ship pre-compiled files with Debian binary packages. So I'm now calling This works for some package like raku-json-name which contains some files in Here's a way to reproduce this problem with rakudo 2022.04 with JSON::Fast.
The verbose output of How can I get this file in the directory specified with All the best |
Did this work on previous versions of rakudo? It sounds like this might be a regression |
I can't say. That's the first time I've packaged JSON::Fast. For what it's worth, I get the same results with |
There have been a lot of changes across those areas of the rakudo code base in the last 6ish months, so it could very well not just be the original install script |
Looks like I'm not the only one: rakudo/rakudo#4907 Sorry, I forgot that this bug report is attached to |
Hi
I need some help to figure out how to best use zef to create Debian package from Perl6 modules.
As Debian packager, my goals are somewhat different from perl6 installer goals:
I think this could be achieved by installing Perl6 modules a bit like emacs extensions are installed:
Can zef support removal of all pre-compiled files ?
If I understand correctly, the pre-compiled files are valid for a single version of rakudo, hence pre-compilation must be redone when a new version of rakudo is installed.
Can you confirm that point ?
If yes, a rakudo upgrade implies to:
Dpkg has mechanisms to trigger an action on all Perl6 packages when rakudo is installed, hence the actions above may be done one by one using a cleanup and recompile step similar to a package upgrade.
I will welcome thoughts or advices before I start to implement Debian packaging for Perl6 modules ;-)
Note that packagers for other distro may have similar needs. Feel free to include them in this discussion to find common ground between Linux distribution with respect to Perl6 packaging.
All the best
The text was updated successfully, but these errors were encountered: