Custom runnable command

[mdelmans]

Hi !

I want to use pants to

1. Build documentation using a python script
2. Publish generated documentation using firebase

I tried the following build

python_source(
	name="generate",
	source="generate.py",
	dependencies=[
		...
	]
)

run_shell_command(
	name="publish",
	command="firebase deploy --only hosting",
	runnable_dependencies=[":generate"]
)

Which doesn't seem to work , pants run /path/to/build:publish does not execute :generate.

Would appreciate if someone can explain what I am doing wrong.

Thank you !

[thejcannon]

IIUC runnable_dependencies just specifies which targets should be placed in the sandbox, in such a way that you could run them. It doesn't run them beforehand. For that you'd need a separate run_shell_command.

The difference is control over args. Pants wouldn't know what to pass to generate, so either you control that (via invoking it yourself and leveraging runnable_dependencies or having Pants invoke it beforehand (leveraging run_shell_command and specifying args in the command)

[huonw]

To add to @thejcannon's answer about the background to runnable_dependencies, it seems like you may be interested in the adhoc_tool target: this target is a good way to get pants to run a "code generation" command for you and capture the output files for later manipulation. For your example, it might look something like:

python_source(
	name="generate",
	source="generate.py",
	dependencies=[
		...
	]
)

adhoc_tool(
    name="generated-docs",
    runnable=[":generate"],
    output_files=["docs.html"], # change this to match what generate.py produces
)

run_shell_command(
	name="publish",
	command="firebase deploy --only hosting",
	execution_dependencies=[":generated-docs"]
)

You'll need to activate the pants.backend.experimental.adhoc backend package in pants.toml too.

It should also be possible to avoid adhoc_tool and have run_shell_command invoke :generate, via runnable_dependencies. However, using adhoc_tool is better, assuming that generate.py is a "pure" process that just produces files (e.g. it doesn't read from external systems like websites that can change, and it doesn't write to external systems). In particular, adhoc_tool runs in a sandbox, can run in parallel with other tasks, and is cached (if generate.py and its dependencies haven't changed, the docs can be served from cache instantly, instead of spending time regenerating the same thing).

[mdelmans]

@thejcannon @huonw thank you so much for the replies!

I think I managed to generate files as a part of :publish and I can see them in the sandbox. But now I have a problem with firebase that can't locate the generated files.

I have a src/py/www folder with BUILD and build.py in it. My BUILD looks like

python_source(
	name="build",
	source="build.py",
	dependencies=[
		...
	]
)

adhoc_tool(
	name="build-docs",
	runnable=":build",
	workdir="",
	output_directories=["src/py/www/public"]
)

run_shell_command(
	name="publish",
	command="firebase deploy --only hosting",
	execution_dependencies=[
		":build-docs"
	]
)

I run:

$ pants run src/py/www:publish

=== Deploying to 'xxx'...

i  deploying hosting

Error: Specified "public" directory "public" does not exist, can't deploy hosting to site "xxx"

For context, firebase deploy expects a public folder to be in the workdir. I can see src/py/www/public folder generated in the sandbox. Am I right that run_shell_command runs in the actual src/py/www directory and doesn't see files in the sandbox ?

Not sure how the path for generated files in build.py is determined (using mkdocs under the hood) but if I run pants run src/py/www:build the public folder appears in src/py/www.

---

😞
I think I'm super confused about all these runnable business, even after reading the docs ten times.

What is the point of runnable_dependecies? They are targets that can be run and we need to run them beforehand but we can't run them because we don't know which args to pass, and need something else to actually run them. executable_dependencies seem to be the same thing, except we actually can (for some reason) execute them beforehand and actually do so. Why have runnable_dependencies at all ?

Or why python_source is runnable (we can do pants run on it) if we can't specify arguments to run it with? And why, on the other hand, adhoc_tool is not runnable (we can't do pants run on it), yet we can pass args to it, and we can use it as an executable_dependency and it will happily run ?

Feels like I'm missing a very critical and important point about how pants works.

[huonw]

For context, firebase deploy expects a public folder to be in the workdir. I can see src/py/www/public folder generated in the sandbox. Am I right that run_shell_command runs in the actual src/py/www directory and doesn't see files in the sandbox ?

Ah, the run_shell_command command actually runs in repository directly, with the execution_dependencies available in a "change root" ({chroot}). This is designed to allow the run_shell_command to edit the repository, e.g. copy files from {chroot} to the repo.

Thus, for your case, you'll need to tell the firebase CLI to look at the {chroot}. Something like:

run_shell_command(
	name="publish",
	command="cd {chroot}/src/py/www && firebase deploy --only hosting",
	execution_dependencies=[
		":build-docs"
	]
)

https://www.pantsbuild.org/docs/reference-run_shell_command

---

Yeah, it's definitely confusing, and I think Pants isn't fully "simplified" yet (the e.g. adhoc_tool feature was only added relatively recently, and is still experimental).

As background, pants generally reasons about files on disk. There's two sorts of files:

• ones directly in the repo like python_sources
• 'built' ones, either "packages" like pex_binary or code-generation that's designed to be used by other targets

For this code, there's three parts:

• python_sources: .py files in the repo. Pants happens to know how to execute them (i.e. set up the appropriate dependencies for python path/to/file.py to work), which is exposed as pants run path/to/file.py. Args can be passed to the file like pants run path/to/file.py -- --some=args here, but there's no way to directly store "please use these args by default when running this Python file", that's more what the other two are for:
• adhoc_tool: a "saved" command that produces files/directories that other targets can use. The adhoc_tool indicates which files are interesting via output_files/output_directories, and other targets use dependencies to "import" them.
• run_shell_command: a "saved" command that executes with useful side effects (e.g. uploading files). This command has two sorts of inputs/dependencies:
  • execution_dependencies: data files that might need to be consumed by the command, e.g. the built HTML files that adhoc_tool outputs, in your example
  • runnable_dependencies: commands/targets that pants manages that the command itself needs to run e.g. in your example, run_shell_command runs the firebase CLI by relying on it being installed on the system, but one could also have a python script or similar in the repo, and then use run_shell_command to run it, by specifying runnable_dependencies=[":some-example-script"] and running some-example-script within the command.

Does that help?

[mdelmans]

Thanks @huonw ! All working now 🥳

Also thanks for the explanation. I makes a bit more sense now. If I think about targets as files it all falls into places a bit neater. Maybe it's all about naming. For example, if adhoc_tool actually represents generated files, maybe it should have been called adhoc_files or generated_files instead. To be consistent with python_sources or pex_binary.

I think I understand the difference between runnable_dependencies and execution_dependencies but it's not crystal clear why we need both and can't just have dependencies instead. If I understand correctly, runnable_dependencies is different because it can take arbitrary commands, like firebase, and not only targets. But in my case, I do not supply firebase as a runnable_dependency and it still works. Are runnable_dependencies and execution_dependencies handled differently by pants ?

P.S.

Sorry for grilling you. I would totally understand if you don't want to spend more time explaining it. I find your explanations useful and maybe some of them can be converted into a PR to documentation pages.

[huonw]

For example, if adhoc_tool actually represents generated files, maybe it should have been called adhoc_files or generated_files instead. To be consistent with python_sources or pex_binary.

Definitely an interesting idea! 👍

I think I understand the difference between runnable_dependencies and execution_dependencies but it's not crystal clear why we need both and can't just have dependencies instead

Yeah, it's a bit subtle.

• A target in execution_dependencies is included as normal files in the shell command's sandbox, so that the shell command can manipulate those files.
• A target in runnable_dependencies is configured as a runnable command, and then the path to that command is added to the shell command's sandbox's $PATH, so that shell command can execute it.

The difference matters for things like Python:

• if you include a python_source as an execution_dependency, the sandbox will include the .py file, but it won't necessarily have the right Python interpreter and so on, so it won't be easy to execute that file. However, the file can still be uploaded to an external system or otherwise manipulated by the shell command, i.e. the source code is treated as data by the command, rather than an executable.
• if you include it as a runnable_dependency, pants will set up the right interpreter and environment to make the source executable. So, the python script is now an executable, not just data.

At least... that's the theory, and how it works for shell_command. Turns out runnable_dependencies doesn't work for run_shell_command: #19731. Thank you for asking questions!

If I understand correctly, runnable_dependencies is different because it can take arbitrary commands, like firebase, and not only targets

Not quite: as above, runnable_dependencies takes a target and (theoretically) adds it to the sandbox's $PATH environment variable so that it can be executed "automatically".

But in my case, I do not supply firebase as a runnable_dependency and it still works.

Yeah, for run_shell_command, I think pants includes your system $PATH, so anything that's available in a normal terminal will also be available for that command. In this case, I imagine you have firebase installed on your system?

Sorry for grilling you. I would totally understand if you don't want to spend more time explaining it. I find your explanations useful and maybe some of them can be converted into a PR to documentation pages.

All good 👍 as you say, these sort of conversations can become documentation... and they find bugs!

[mdelmans]

Thanks again for the explanation. I think I get the difference a bit better now. Will put it to test next time I write a new BUILD file.

It's again a problem of naming for me. 'execution' and 'runnable' have a very similar meaning so the difference between execution_dependency and runnable_dependency is not intuitive.

From your own words, execution_dependency is treated as data, rather than an executable 😅. Which kind-of makes sense: execution_dependency is something we need for execution but itself is not executable. But from now on, I will translate execution_dependency to data_dependency or input_dependency in my head 😄

[mdelmans]

@huonw have issues with a variation on the topic.

This time I generate some C code I want to use from Python. I will illustrate the issue with a simpler example.

I have:

/src/py/gen_test/gen.py

import os

code = """
def my_sum(a: int, b: int):
    return a + b
"""

os.makedirs("build")

with open("build/sum.py", "w") as f:
    f.write(code)

/src/py/gen_test/run.py

from .build.sum import my_sum

if __name__ == "__main__":
    print(my_sum(1, 1))

and /src/py/gen_test/BUILD

file(
    name="gen",
    source="gen.py"
)

shell_command(
    name="build",
    command="python gen.py",
    output_dependencies=[
       ":gen"
    ],
    tools=["python"],
    output_directories=["build/"],
)

python_source(
    name="run",
    source="run.py",
    dependencies=[
        ":build"
    ]
)

pex_binary(
    name="run-pex",
    entry_point="run.py",
    dependencies=[
        ":build"
    ]
)

I can do pants run src/py/gen_test:run no problem but pants run src/py/gen_test:run-pex complains with:

  File "/Users/mdp/.pex/unzipped_pexes/e1c0227cf16eb3a5fa0391eb6f11a210182a14a4/gen_test/run.py", line 1, in <module>
    from .build.sum import my_sum
ModuleNotFoundError: No module named 'gen_test.build'

I tried using experimental_wrap_as_resources but with the same result

experimental_wrap_as_resources(
    name="resource-build",
    inputs=[":build"],
)
pex_binary(
    name="run-pex",
    entry_point="run.py",
    dependencies=[
        ":resource-build"
    ]
)

Any suggestions ?

[huonw]

If you can create a reduced example I can run on my machine, I'll be able to provide specific feedback/debugging. Otherwise, I'd suggest starting with running pants package on the pex and inspecting it (it's a zip file under the hood, just has a .pex extension) to see what's inside. For example: validate that the files end up in the expected places etc

[mdelmans]

@huonw thanks !

I'm not sure I can reduce it further, it is pretty minimal.

I followed your advice with pants package and found the following directory structure:

gen_test/
├─ run.py
src/
├─ py/
│  ├─ gen_test/
│  │  ├─ build/
│  │  │  ├─ sum.py

So it actually generates the files, but for some reason puts them into src/py/gen_test instead of gen_test inside the pex.

Is it a bug ?

[mdelmans]

Gist here. The names are a bit different but the behaviour is the same.

[huonw]

Thanks for the gist!

Unfortunately, I couldn't reproduce the behaviour. The //:build didn't run directly on my machine (e.g. the Python interpreters are a bit weird), so I switched the internals to construct the file just with bash directly, but this shouldn't affect the downstream behaviour: once it's generated, they're just files (I hope). I also added a pants.toml and updated the import from from .lib.sum import ... to from lib.sum import ... (the relative import with the . didn't work).

I put those edits in https://gist.github.com/huonw/200f6a77fde73ab26f71644f179bfc1a .

When I run pants run :run-pex with that code, the PEX builds and runs successfully, printing 2. And pants package :run-pex followed by unzip -l dist/run-pex.pex shows that the file is at the expected location lib/sum.py.

git clone https://gist.github.com/huonw/200f6a77fde73ab26f71644f179bfc1a
cd 200f6a77fde73ab26f71644f179bfc1a
pants run :run-pex # prints `2`
pants package :run-pex
unzip -l dist/run-pex.pex | grep sum # prints `... lib/sum.py`