Build `scie-pants` using `science`. #191

jsirois · 2023-05-26T02:51:10Z

Instead of using raw scie-jump and a hand-crafted JSON lift manifest,
use science and a TOML lift manifest to build scie-pants.

The science tool and its more friendly interface are the present and
future of end-user scie construction. We convert here and gain a few
things:

A manifest format we can comment.
Abstraction over platforms and PBS details.
Support for embedded build provenanace information for more
comprehensive black box debugability.
A clean path to publishing both "thin" and "fat" binaries, solving
Option to avoid download python exe from https://github.com/indygreg/python-build-standalone #154.

Instead of using raw `scie-jump` and a hand-crafted JSON lift manifest, use `science` and a TOML lift mabifest to build `scie-pants`. The `science` tool and its more friendly interface are the present and future of end-user scie construction. We convert here and gain a few things: + A manifest format we can comment. + Abstraction over platforms and PBS details. + Support for embedded build provenanace information for more comprehensive black box debugability. + A clean path to publishing both "thin" and "fat" binaries and solving pantsbuild#154.

jsirois · 2023-05-26T02:55:15Z

N.B.: I won't land this until science 0.1.2 is available (See: a-scie/lift#21).
With that, the lift manifest build info is like so:

$ SCIE=inspect dist/scie-pants-linux-x86_64 | jq .science
{
  "app_info": {
    "repo": "https://github.com/pantsbuild/scie-pants"
  },
  "binary": {
    "hash": "8b2a9743af6957d49b5b501a50a1a92ccf5aebcd575261127e457b10294814b8",
    "size": 6468367,
    "url": "https://github.com/a-scie/lift/releases/download/v0.1.2/science-linux-x86_64",
    "version": "0.1.2"
  },
  "git_state": "v0.7.2-4-g0943208-dirty",
  "manifest": {
    "hash": "5b79b2d2055380116e8cbf684f230f3af1f816894d16765a8b5bde8ddd19b379",
    "size": 5300,
    "source": "package/scie-pants.toml"
  },
  "notes": [
    "This scie lift JSON manifest was generated from a source lift toml manifest using the science binary.",
    "Find out more here: https://github.com/a-scie/lift/blob/v0.1.2/README.md"
  ]
}

And the zip-trailer is preserved:

$ cat <(unzip -l dist/scie-pants-linux-x86_64 | head) <(echo ...) <(unzip -l dist/scie-pants-linux-x86_64 | tail)
warning [dist/scie-pants-linux-x86_64]:  6308880 extra bytes at beginning or within zipfile
  (attempting to process anyway)
Archive:  dist/scie-pants-linux-x86_64
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  1980-01-01 00:00   .bootstrap/
        0  1980-01-01 00:00   .bootstrap/pex/
      401  1980-01-01 00:00   .bootstrap/pex/__init__.py
      235  1980-01-01 00:00   .bootstrap/pex/__main__.py
      601  1980-01-01 00:00   .bootstrap/pex/argparse.py
    11259  1980-01-01 00:00   .bootstrap/pex/atomic_directory.py
      483  1980-01-01 00:00   .bootstrap/pex/attrs.py
warning [dist/scie-pants-linux-x86_64]:  6308880 extra bytes at beginning or within zipfile
  (attempting to process anyway)
...
     4897  1980-01-01 00:00   scie_pants/install_pants.py
     1680  1980-01-01 00:00   scie_pants/log.py
    43256  1980-01-01 00:00   scie_pants/pants_release_tags.json
     6665  1980-01-01 00:00   scie_pants/pants_version.py
     1601  1980-01-01 00:00   scie_pants/ptex.py
     1474  1980-01-01 00:00   scie_pants/record_scie_pants_info.py
      346  1980-01-01 00:00   scie_pants/scie_pants.iml
     9605  1980-01-01 00:00   scie_pants/update_scie_pants.py
---------                     -------
  4047731                     386 files

jsirois · 2023-05-26T02:59:38Z

package/scie-pants.toml

+lazy = true
+version = "3.8.16"
+
+[[lift.interpreters]]


N.B.: Adding 3.11 support just requires adding a similar table with "3.11.3" as the version, an id of "cpython311" and an edit to tools/src/scie_pants/configure_pants.py that selects "cpython311" for the appropriate Pants version cutoff.

jsirois · 2023-05-26T03:23:11Z

Alright, some more work is needed here for the Linux CI case which uses a volume mount into an alpine linux container: there are now paths not mounted. There is also investigation needed to find the root of the Mac OS x86_64 issue.

sureshjoshi · 2023-05-26T13:35:46Z

I'd seen the error thrown by the x86 Mac, which happened when I was using the wrong release name for the cpythons (I think I used version, instead of 20230507). Not the problem in the toml though, so I ran what I think is the correct CI command on my spare x86 Mac mini to see if it gave any better information, but I get a different error entirely

Note: Haven't even started to debug or confirmed scie-pants usage on my machine yet (just provisioned it this morning). Will try to debug a bit more this evening when I'm home

RUST_BACKTRACE=full cargo run -p package -- test --check --tools-pex-mismatch-warn

>> Verify PANTS_SOURCE mode.
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint: 
hint: 	git config --global init.defaultBranch <name>
hint: 
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint: 
hint: 	git branch -m <name>
Initialized empty Git repository in /private/var/folders/vl/4zxxh40x09d4m104x01jk1p80000gn/T/.tmpoDxIim/.git/
remote: Enumerating objects: 2130, done.
remote: Counting objects: 100% (2130/2130), done.
remote: Compressing objects: 100% (1865/1865), done.
remote: Total 2130 (delta 410), reused 689 (delta 208), pack-reused 0
Receiving objects: 100% (2130/2130), 3.45 MiB | 11.22 MiB/s, done.
Resolving deltas: 100% (410/410), done.
From https://github.com/pantsbuild/pants
 * branch            cfcb23a97434405a22537e584a0f4f26b4f2993b -> FETCH_HEAD
HEAD is now at 33a6cf8 Version bump to 2.14.1.
The Pants 2.14.1 clone is working.
Pants from sources argv: -V.
/private/var/folders/vl/4zxxh40x09d4m104x01jk1p80000gn/T/.tmpoDxIim/build-support/pants_venv: line 22: -c: command not found
Bootstrapping pants_deps with requirements:
  /private/var/folders/vl/4zxxh40x09d4m104x01jk1p80000gn/T/.tmpoDxIim/3rdparty/python/requirements.txt
/private/var/folders/vl/4zxxh40x09d4m104x01jk1p80000gn/T/.tmpoDxIim/build-support/pants_venv: line 22: -c: command not found
/private/var/folders/vl/4zxxh40x09d4m104x01jk1p80000gn/T/.tmpoDxIim/build-support/pants_venv: line 22: -c: command not found
/private/var/folders/vl/4zxxh40x09d4m104x01jk1p80000gn/T/.tmpoDxIim/build-support/pants_venv: line 35: : command not found

Failed to create venv.
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Command cd "/var/folders/vl/4zxxh40x09d4m104x01jk1p80000gn/T/.tmpoDxIim" && PANTS_VENV_DIR_PREFIX="/var/folders/vl/4zxxh40x09d4m104x01jk1p80000gn/T/.tmp0GGzQ0" "./pants" "-V" failed with exit code: Some(1)
STDOUT not captured.
STDERR not captured.', package/src/test.rs:516:10
stack backtrace:
   0:        0x10f79d976 - std::backtrace_rs::backtrace::libunwind::trace::h863be41b53c4f884
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:        0x10f79d976 - std::backtrace_rs::backtrace::trace_unsynchronized::ha57069b30dcbe3f5
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:        0x10f79d976 - std::sys_common::backtrace::_print_fmt::h7dbf0829c4d9fd22
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/sys_common/backtrace.rs:65:5
   3:        0x10f79d976 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::he84dde8223670de4
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/sys_common/backtrace.rs:44:22
   4:        0x10f7b91aa - core::fmt::write::h0721d4325d1d86bb
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/core/src/fmt/mod.rs:1232:17
   5:        0x10f79a89c - std::io::Write::write_fmt::hf4d04b4d333539fa
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/io/mod.rs:1684:15
   6:        0x10f79d75a - std::sys_common::backtrace::_print::h15efcfdd13c9c026
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/sys_common/backtrace.rs:47:5
   7:        0x10f79d75a - std::sys_common::backtrace::print::hed06273df3ccacbe
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/sys_common/backtrace.rs:34:9
   8:        0x10f79f323 - std::panicking::default_hook::{{closure}}::he8c58e8d0abf03c5
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panicking.rs:271:22
   9:        0x10f79f078 - std::panicking::default_hook::h8320561bf55c0290
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panicking.rs:290:9
  10:        0x10f79f90b - std::panicking::rust_panic_with_hook::h38dc40cfa47552ba
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panicking.rs:692:13
  11:        0x10f79f824 - std::panicking::begin_panic_handler::{{closure}}::h727c67dc750b5cee
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panicking.rs:583:13
  12:        0x10f79ddb9 - std::sys_common::backtrace::__rust_end_short_backtrace::h6848121e0bfe63be
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/sys_common/backtrace.rs:150:18
  13:        0x10f79f4ed - rust_begin_unwind
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panicking.rs:579:5
  14:        0x10f7cc013 - core::panicking::panic_fmt::h690b01655a033c26
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/core/src/panicking.rs:64:14
  15:        0x10f7cc435 - core::result::unwrap_failed::h318e014235203f60
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/core/src/result.rs:1750:5
  16:        0x10f4707c8 - core::result::Result<T,E>::unwrap::h591139f0439cf875
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/core/src/result.rs:1090:23
  17:        0x10f443f35 - package::test::test_pants_source_mode::h66615db42bee4e92
                               at /Users/sj/Developer/oss/scie-pants/jpants/package/src/test.rs:510:9
  18:        0x10f43d8b0 - package::test::run_integration_tests::h9510177f8e6ddc5c
                               at /Users/sj/Developer/oss/scie-pants/jpants/package/src/test.rs:114:9
  19:        0x10f42fdf9 - package::maybe_build::h8c567072a88a17dc
                               at /Users/sj/Developer/oss/scie-pants/jpants/package/src/main.rs:202:13
  20:        0x10f430a8a - package::main::h1e24756d9761c18c
                               at /Users/sj/Developer/oss/scie-pants/jpants/package/src/main.rs:252:31
  21:        0x10f438ede - core::ops::function::FnOnce::call_once::h08c4734045481dad
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/core/src/ops/function.rs:250:5
  22:        0x10f45c081 - std::sys_common::backtrace::__rust_begin_short_backtrace::h4e853009c755484a
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/sys_common/backtrace.rs:134:18
  23:        0x10f46a2a4 - std::rt::lang_start::{{closure}}::hf2ee41888ae36a6c
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/rt.rs:166:18
  24:        0x10f794730 - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::hb633d246db9c8b3f
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/core/src/ops/function.rs:287:13
  25:        0x10f794730 - std::panicking::try::do_call::hcc14a5a158bf6961
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panicking.rs:487:40
  26:        0x10f794730 - std::panicking::try::ha053de4a5d927432
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panicking.rs:451:19
  27:        0x10f794730 - std::panic::catch_unwind::hdc9235c152d51b66
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panic.rs:140:14
  28:        0x10f794730 - std::rt::lang_start_internal::{{closure}}::h77107745b5529819
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/rt.rs:148:48
  29:        0x10f794730 - std::panicking::try::do_call::h9edf685df72863ef
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panicking.rs:487:40
  30:        0x10f794730 - std::panicking::try::h2dfb970c45e23c24
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panicking.rs:451:19
  31:        0x10f794730 - std::panic::catch_unwind::hc60a4924b5d35df8
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panic.rs:140:14
  32:        0x10f794730 - std::rt::lang_start_internal::h733b9f59d292f4af
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/rt.rs:148:20
  33:        0x10f46a277 - std::rt::lang_start::hcb7336cb466390df
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/rt.rs:165:17
  34:        0x10f434418 - _main
  35:     0x7ff8188b441f - <unknown>

                    'c.          sj
                 ,xNMM.          ------------------- 
               .OMMMMo           OS: macOS 13.4 22F66 x86_64 
               OMMM0,            Host: Macmini8,1 
     .;loddo:' loolloddol;.      Kernel: 22.5.0 
   cKMMMMMMMMMMNWMMMMMMMMMM0:    Uptime: 23 hours, 55 mins 
 .KMMMMMMMMMMMMMMMMMMMMMMMWd.    Packages: 21 (brew) 
 XMMMMMMMMMMMMMMMMMMMMMMMX.      Shell: zsh 5.9 
;MMMMMMMMMMMMMMMMMMMMMMMM:       Resolution: 1920x1080 
:MMMMMMMMMMMMMMMMMMMMMMMM:       DE: Aqua 
.MMMMMMMMMMMMMMMMMMMMMMMMX.      WM: Quartz Compositor 
 kMMMMMMMMMMMMMMMMMMMMMMMMWd.    WM Theme: Blue (Dark) 
 .XMMMMMMMMMMMMMMMMMMMMMMMMMMk   Terminal: /dev/ttys000 
  .XMMMMMMMMMMMMMMMMMMMMMMMMK.   CPU: Intel i5-8500B (6) @ 3.00GHz 
    kMMMMMMMMMMMMMMMMMMMMMMd     GPU: Intel UHD Graphics 630 
     ;KMMMMMMMWXXWMMMMMMMk.      Memory: 9903MiB / 32768MiB 
       .cooc,.    .,coo:.

jsirois · 2023-05-26T14:40:58Z

@sureshjoshi yeah, your mac mini error is in Pants, not scie-pants. It's trying to run Pants from sources and getting an error in the build-support/pants_venv script. It looks like $PY is empty here: https://github.com/pantsbuild/pants/blob/release_2.14.1/build-support/pants_venv#L20; so your mini does not have a Python 3.{7,8,9} on the PATH.

jsirois · 2023-05-26T16:02:34Z

Aha, OK - the Linux arm issue is specifically the issue that PBS for Linux + arm + musl does not exist as a thing, just Linux + arm + glibc and Linux + x86_64 + {glibc,musl} are things. The final scie build will need to be back on the other side of the alpine image build of the scie-pants Rust binary.

stuhood · 2023-05-26T17:26:26Z

package/src/main.rs

@@ -204,45 +211,102 @@ fn maybe_build(args: &Args, build_context: &BuildContext) -> Result<Option<PathB
            check,
            tools_pex_mismatch_warn,
        } => {


It seems like the four branches of Commands::Test (and Commands::Scie) could be unified into one branch which initialized the tools_pex or scie_pants if they weren't set? I know this was here before, but as each branch gets longer (in very similar ways) the (apparent?) duplication is more obvious.

Yeah, agreed. I didn't want to fiddle in this PR.

Done here: #194

This was noted tech debt in the conversion from raw scie-jump to science PR here: #191 (comment)

jsirois requested review from stuhood, kaos, benjyw and sureshjoshi May 26, 2023 02:51

jsirois mentioned this pull request May 26, 2023

Fix ptex file order and science url in build info. a-scie/lift#21

Merged

jsirois commented May 26, 2023

View reviewed changes

Fix up tools.pex sharing over volume mounts.

40188a8

Split up scie-pants build from scie build for musl.

e0370f3

jsirois force-pushed the issues/189 branch from ee2581c to e0370f3 Compare May 26, 2023 16:42

Upgrade to science 0.1.2.

6725632

stuhood approved these changes May 26, 2023

View reviewed changes

jsirois merged commit 0ebe82a into pantsbuild:main May 26, 2023
5 checks passed

jsirois deleted the issues/189 branch May 26, 2023 17:44

This was referenced May 27, 2023

Convert the scie-pants binary manifest / building to science #189

Closed

DRY up the package tool main. #194

Merged

jsirois added a commit that referenced this pull request May 28, 2023

DRY up the package tool main. (#194)

686b544

This was noted tech debt in the conversion from raw scie-jump to science PR here: #191 (comment)

kaos mentioned this pull request Jan 22, 2024

Prep support for running Pants with Python 3.10 or 3.11. #351

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build `scie-pants` using `science`. #191

Build `scie-pants` using `science`. #191

jsirois commented May 26, 2023 •

edited

Loading

jsirois commented May 26, 2023

jsirois May 26, 2023

jsirois commented May 26, 2023

sureshjoshi commented May 26, 2023

jsirois commented May 26, 2023 •

edited

Loading

jsirois commented May 26, 2023

stuhood May 26, 2023

jsirois May 26, 2023

jsirois May 27, 2023

Build scie-pants using science. #191

Build scie-pants using science. #191

Conversation

jsirois commented May 26, 2023 • edited Loading

jsirois commented May 26, 2023

jsirois May 26, 2023

Choose a reason for hiding this comment

jsirois commented May 26, 2023

sureshjoshi commented May 26, 2023

jsirois commented May 26, 2023 • edited Loading

jsirois commented May 26, 2023

stuhood May 26, 2023

Choose a reason for hiding this comment

jsirois May 26, 2023

Choose a reason for hiding this comment

jsirois May 27, 2023

Choose a reason for hiding this comment

Build `scie-pants` using `science`. #191

Build `scie-pants` using `science`. #191

jsirois commented May 26, 2023 •

edited

Loading

jsirois commented May 26, 2023 •

edited

Loading