How to create a package for an AWS lambda?

[tmccombs]

I'm posting this as a discussion in case there is a good solution I missed, but it's also kind of a feature request if there isn't a good solution.

I want to create a zip file for a python package for AWS lambda. For this to work as expected, it would need to have a layout that looks something like:

root
 |
 |-- my_handler.py
 | -- dependency1/
 | -- dependency2/

or even

root
|
|-- my_package/
  |
  | -- my_handler.py
|-- dependency1/
|-- dependency2/

That shouldn't be too hard to get right?

Nope.

It is surprisingly difficult to get a zip file in this format.

If I use py_binary and use the "python_zip_file" output group (which by the way is not documented anywhere), then my sources are under the full path from the bazel root. And the dependencies are under paths that are prefixed with the name of the pip_parse rule. And there is a separate launcher python script that is unneeded.

If I create a wheel (and again, I couldn't find any documentation outside of the actual source code), I still have the full path to the package in the output. But at least the dependencies are somewhat more logical. Although they are in a site_pacakges directory, which means they won't load correctly when the lambda executes.

I tried using pkg_zip, but in order for that to work, I need to explicitly list every transitive dependency, with appropriate strip_prefix rules, which isn't very practical.

The most realistic solution right now is something like https://stackoverflow.com/a/72049294/2543666

But that is pretty complicated, and involves

Is there really not a better way to do this?

[tmccombs]

Looking at the stack overflow question in further detail, it doesn't handle well, the case where you pull in another non-external python library with a different package.

[aignas]

Right now there is no good way to do this and I am not sure if this is something that rules_python should implement. This could be better done as a separate ruleset that wraps the complexity in a reusable way.

What you are asking probably needs to be implemented as a build rule and it needs to inspect the PyInfo providers to get the output structure right.

[tmccombs]

The problem is that PyInfo doesn't provide the necessary information (at least as far as I can tell). Maybe it is just my lack of knowedge about bazel and rules_python, but PyInfo only seems to expose information about individual files that are dependencies, but I need to know about what packages, not just individual files.

I did eventually get something working but:

• It had to make some assumptions about the layout of code in my project, that probably wouldn't be appropriate in a public re-usable library
• It had to transform the path of every file in the transitive dependencies, which is made even more difficult by the fact that bazel doesn't have a way to do a regex replace (see Any plan on adding some support for regular expressions to the language? google/starlark-go#241)
• I have to strip out the site-packages/__init__.py file, which I'm not entirely sure if that is always safe to do.

What would be helpful is if PyInfo had way to get a list of transitive package dependencies, where each package has a list of files in the package, with a way to get the relative path of each file, to the package.