RFC: FPM packaging #484

zyrolasting · 2022-03-02T19:18:21Z

Package PaSh for system-level package managers using FPM

Update: 2022 March 10

I learned that macOS is not supported a bit late, so I'm stopping macOS work. I need acceptance criteria for this PR, so that I know what to do before this is ready to merge.

Update: 2022 March 7

Fedora working (without dictionary data) as of ee36b6b

Starting macOS. This will be the last supported platform on my end.

Update: 2022 March 4

Debian, Ubuntu, and Pop_OS! are working. Fedora has a problem with the the mv hack for Python site-packages and dist-packages. Arch lacks a dictionary for the intro example, but pa.sh runs.

This one is fun to review with tmux.

cd scripts/package
./iterate.sh ubuntu deb
./iterate.sh archlinux pacman
./iterate.sh fedora rpm

The system obsoletes many deployment scripts, but that doesn't really matter until packages are hosted somewhere.

Please review and let me know what you'd need to merge this as a feature.

I see some distribution inference rules that seemed to pop up several times. The script seems to be useful in several places, but I have not deduplicated the code yet. That would involve changes that are out of scope.

Deployment would be easier if `pa.sh` helped users with `PASH_TOP` and a default `~/.pash_init`. I saw scripts to modify the user's rcfiles (which is not ideal, imo). With process substitution, the process for setting `PASH_TOP` is a little nicer. $ cd "$(find ~ -type f pa.sh | xargs -n 1 dirname | head -n 1)" $ ./pa.sh # Add this to your shell configuration. export PASH_TOP='/home/sage/prj/pash' export PATH="$PASH_TOP:$PATH" # If your shell supports process substitution (bash, zsh, etc.) # run `. <(./pa.sh)` to use PaSh in this shell. The key benefit for deployment is that you can simply instruct a user to actually start using PaSh and follow its instructions, as opposed to adding installation steps. Since this is Bash, I use `return` to prevent accidental execution of further instructions when `pa.sh` is sourced.

Some code only calls `make` with no target specified. No `autogen.sh`, `configure`, etc. Maybe that's fine for the runtime, but it's not enough for the `libdash` derivative. I wonder if there's a different script that I haven't seen yet that does this better.

zyrolasting · 2022-03-02T19:28:48Z

Question: With C in mind, what target architectures are we talking about? I'm assuming noarch for the packages, but I don't know if I should.

angelhof

Looks good so far, left one comment. Tagging Michael (@mgree ) who is working on simplifying the installation of libdash, since you mentioned that you are struggling with this.

I think that Michael is planning to have a working clean installer for libdash by the end of March. Michael, am I saying this right?

angelhof · 2022-03-02T19:48:53Z

pa.sh

+
+
+rcfile=~/.pash_init
+if [ ! -f "$rcfile" ]; then


Do we want to do those everytime? Or maybe just with a pash-init.sh script? I am afraid that they will add to the overhead of the shell script. Also, I don't think we should print to stdout (see 3 lines below).

EDIT: This code is here because it's not suited for system-level deployment. Is it better to use a command line flag here, or ship a configuration helper?

Configuration helper is likely the way to go (for now we can have it in setup-pash.sh?), as we're trying to minimize startup overheads.

Configuration helper is likely the way to go (for now we can have it in setup-pash.sh?), as we're trying to minimize startup overheads.

setup-pash.sh is not suitable for post-install steps because it mixes user-level and system-level commands, and starts from the user's perspective. If I did the same:

$ sudo apt install pash # ~ is not correct. $ pa.sh # Not configured for the user

Putting the code in pa.sh was meant to avoid adding an additional setup command, but shell configuration must happen on the user's command. I was going for this:

$ sudo apt install pash # Installs $ pa.sh # Asks for configuration initially, then helps

mgree · 2022-03-02T19:52:45Z

Yes---my plan is to have something like pip install libdash put appropriate Python modules in the right place.

We would want to change setup.sh to use that new process, storing things wherever we're keeping the other vendored packages.

zyrolasting · 2022-03-02T19:57:53Z

Yes---my plan is to have something like pip install libdash put appropriate Python modules in the right place.

We would want to change setup.sh to use that new process, storing things wherever we're keeping the other vendored packages.

Gotcha. I'm doing something different. Here, PaSh is a system level package, so pip is invoked as part of setup. You can see in the code how I tell FPM to prepare packages via /bin/python3 -m pip install. I can just as easily invoke your system in a post-install script, so we could collaborate there.

mgree · 2022-03-02T20:01:05Z

Last I heard anything about it, PaSh was vendoring dependencies in its own directory rather than installing on the system properly. My plan is to make libdash a conventional pip package, rather than a submodule of the pash repo. If PaSh is moving to a more conventional, system-package model, then great, it'll make everything easier---especially if setup is already using pip.

zyrolasting · 2022-03-02T20:01:59Z

Last I heard anything about it, PaSh was vendoring dependencies in its own directory rather than installing on the system properly. My plan is to make libdash a conventional pip package, rather than a submodule of the pash repo. If PaSh is moving to a more conventional, system-package model, then great, it'll make everything easier---especially if setup is already using pip.

What you are making is fantastic, because we can simply list it in requirements.txt. Can we find a way to merge our powers? :)

zyrolasting · 2022-03-02T20:02:58Z

And yes, you are correct, the Python packages are not properly set up. It all comes from difficulty organizing PaSh for use as a Python package itself. Moving to system packages solves a lot of problems you're having, and brings the focus towards distribution-specifics.

scripts/package/shell.sh

scripts/package/tools/build

zyrolasting · 2022-03-03T02:12:25Z

Got libdash compilation working on my machine in 3191b89

mgree · 2022-03-03T02:17:42Z

Nice! That means porting to my stuff should be easy, when it's ready.

My cleanup work is back on the original repo, at mgree/libdash on branch python-bindings. I don't have the setup.py/pyproject.toml written up yet.

zyrolasting · 2022-06-15T16:49:34Z

Merge conflicts fixed, reopening for discussion and testing in light of new changes. Any comment I did not address here was addressed in an email, so please link me to any questions you'd like me to address here specifically. I'm removing pash-init.sh as a consequence of a discussion early in the PR's life.

zyrolasting · 2022-06-15T17:27:46Z

Repeating the commands from the docs used to build and test the packages.

version=0.0.1
PATH="$PASH_TOP/scripts/package:$PATH"
deploy.sh archlinux $version pacman
deploy.sh debian $version deb
deploy.sh ubuntu $version deb
deploy.sh fedora $version rpm

Good news: No major change in behavior. Fedora and Arch launch PaSh successfully but without dictionary data (due to lack of wamerican-insane IIRC). Ubuntu and Debian run evaluation/intro/hello-world.sh without issue.

I still see the behavior @dkarnikis explains in #484 (comment) . I think this is out of scope for merging.

zyrolasting · 2022-06-15T17:36:30Z

I manually approved the DCO sign-off, since A) I didn't know of the requirement until reading the details now, and B) I could certify I wrote the commits here with my manual approval to avoid a long rebase. If you'd rather I amend all of the commits, please let me know.

I've been told that fixing `pa.sh`'s fallback value for `PASH_TOP` would cause performance penalties. This patch updates `PASH_TOP` in `pa.sh` as a post-installation step for packages Verified with existing Docker images, but do not assume 100% compatibility with all package managers. Unsure if there are special rules out there that prevent an installer writing to a file extracted from a package.

Add GitHub Actions paragraph content

zyrolasting added 18 commits March 1, 2022 19:12

Add requirements.txt

d3b956b

Draft documentation for upcoming changes

71bf2ea

Add scripts/distro.sh

5911c01

I see some distribution inference rules that seemed to pop up several times. The script seems to be useful in several places, but I have not deduplicated the code yet. That would involve changes that are out of scope.

Draft package build system

3a8d913

Use .pash_init, not .pash-init

07b3676

Allow shell options for container front end

cec3547

Remove some after-install.sh errors regarding printf and ln

102b616

Exclude '*.git*' and clobber built packages

30c3e44

Mark distro.sh executable

170bd2d

Improve config preview

47a4a94

Rephrase Docker volumes

24abb5d

Exclude package build output from future packages

39ab2f8

Use PASH_TOP in distro.sh

91851e4

Exclude python cache files in package builds

7ff2ae0

Remove --epoch due to incorrect Debian package specifier

73f04a2

Stop trying to be clever with previews

58be73e

angelhof reviewed Mar 2, 2022

View reviewed changes

zyrolasting commented Mar 2, 2022

View reviewed changes

scripts/package/shell.sh Outdated Show resolved Hide resolved

scripts/package/tools/build Show resolved Hide resolved

scripts/package/tools/build Show resolved Hide resolved

scripts/package/tools/build Show resolved Hide resolved

zyrolasting mentioned this pull request Mar 2, 2022

Setuptools pash zyrolasting/pash#1

Open

Compile libdash

3191b89

Remove pash-init.sh

42f85dc

zyrolasting requested review from dkarnikis and nvasilakis June 15, 2022 17:47

zyrolasting added 24 commits June 23, 2022 15:12

Remove dead code

0875402

Add package.yaml workflow

d180d2f

Prepare for GitHub Actions

20179f1

Try all branches for gui packaging workflow

eb629de

Use different wd

bd8a9dd

Use consistent working directory path for make

e7966b3

Try removing --squash from fpm Makefile

e3c482d

Upgrade docker login to v2 and don't logout at end of job

6c241f1

Try manual login in deploy.sh

02019b4

Delete trailing whitespace

c1c4683

Alternative workflow

d042c5f

Use the right workflow

2671229

Fix docker command in package-build.yaml

ef6e728

Add --username

bdf6367

Circulate docker credentials

ec7a45c

Remove --squash

4511bd3

Fix yaml

1931a48

Remove excess quotes

1de0316

Prep for shipment

5131b59

Update packaging.md

41fb1a9

Add GitHub Actions paragraph content

Add warning

0ba9784

Reword docs

e30d33a

Use sed for both clone points

f61e80f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RFC: FPM packaging #484

RFC: FPM packaging #484

zyrolasting commented Mar 2, 2022 •

edited

zyrolasting commented Mar 2, 2022 •

edited

angelhof left a comment

angelhof Mar 2, 2022

zyrolasting Mar 5, 2022

nvasilakis Mar 5, 2022

zyrolasting Mar 7, 2022 •

edited

mgree commented Mar 2, 2022

zyrolasting commented Mar 2, 2022 •

edited

mgree commented Mar 2, 2022

zyrolasting commented Mar 2, 2022

zyrolasting commented Mar 2, 2022 •

edited

zyrolasting commented Mar 3, 2022

mgree commented Mar 3, 2022

zyrolasting commented Jun 15, 2022 •

edited

zyrolasting commented Jun 15, 2022 •

edited

zyrolasting commented Jun 15, 2022 •

edited

RFC: FPM packaging #484

Are you sure you want to change the base?

RFC: FPM packaging #484

Conversation

zyrolasting commented Mar 2, 2022 • edited

Update: 2022 March 10

Update: 2022 March 7

Update: 2022 March 4

zyrolasting commented Mar 2, 2022 • edited

angelhof left a comment

Choose a reason for hiding this comment

angelhof Mar 2, 2022

Choose a reason for hiding this comment

zyrolasting Mar 5, 2022

Choose a reason for hiding this comment

nvasilakis Mar 5, 2022

Choose a reason for hiding this comment

zyrolasting Mar 7, 2022 • edited

Choose a reason for hiding this comment

mgree commented Mar 2, 2022

zyrolasting commented Mar 2, 2022 • edited

mgree commented Mar 2, 2022

zyrolasting commented Mar 2, 2022

zyrolasting commented Mar 2, 2022 • edited

zyrolasting commented Mar 3, 2022

mgree commented Mar 3, 2022

zyrolasting commented Jun 15, 2022 • edited

zyrolasting commented Jun 15, 2022 • edited

zyrolasting commented Jun 15, 2022 • edited

zyrolasting commented Mar 2, 2022 •

edited

zyrolasting commented Mar 2, 2022 •

edited

zyrolasting Mar 7, 2022 •

edited

zyrolasting commented Mar 2, 2022 •

edited

zyrolasting commented Mar 2, 2022 •

edited

zyrolasting commented Jun 15, 2022 •

edited

zyrolasting commented Jun 15, 2022 •

edited

zyrolasting commented Jun 15, 2022 •

edited