Shorthand for including an output file of a genrule directory as a src

[mikekap]

If you have a genrule that creates a directory with several files, it's a bit of a pain to write another genrule rule for each (expected) output, just to be able to pass it to *_library (java, python, etc). It would be nice to allow you to somehow specify this in the sources - maybe via flavors (although it is a bit hackish)? e.g.

genrule(
    name='gen_my_thing',
    out='things',
    cmd='$(exe :my_custom_generator) $SRCDIR $OUT',
    srcs=glob(['File1.java', 'File2.java']),
)

java_library(
    name='lib',
    srcs=[
        ':gen_my_thing#lib_output/File1Out.java',
        ':gen_my_thing#lib_output/File2Out.java',
    ]
)

[sdwilsh]

If we were to support this, we'd have to declare each and every output. We have a hacky way we did this internally that basically redefines genrule to be something that generates a genrule for each output.

[sdwilsh]

Internally, that'd probably create an export_file rule addressable by the flavor like you have there.

cc @shs96c and @Coneko for design feedback there, but I think we'd take a PR for this if you wanted to make it official.

[Coneko]

We've discussed this before and we want the syntax to be with brackets like rule[file].

We won't get around implementing it anytime soon though.

[sdwilsh]

@mikekap, if you want to take a stab at this, I think we've given you a bunch of design direction. If not, that's okay too. :)

[njlr]

Has anyone made any progress on this?

I am having a similar issue where a legacy build system generates a folder of headers that I need to reference. Unfortunately the legacy build system is very complex, and reimplementing it properly in Buck is probably unrealistic, hence the need to "wrap" it.

[Coneko]

Note this is just a UI improvement, it would still be functionally equivalent to creating a genrule for each file, minus the copy.

[njlr]

@Coneko Thanks for the response. Is there a recommended approach when there are hundreds of files? I don't think it's possible to read the output directory of the make genrule to get the list of header files.

[Coneko]

You need to know the names of the files in advance and write them in the build files.

You can do that using a python loop if you wish, but you cannot list files in the output directory.

The same limitation exists with the new syntax.

[njlr]

Yes, that works.

For those who stumble on this, try something like:

def extract(x):
  genrule(
    name = x,
    out = x,
    cmd = 'cp $(location :make)/install/include/' + x ' $OUT',
  )
  return ':' + x

headers = [
  'generated_header.h',
]

prebuilt_cxx_library(
  name = 'my-lib',
  exported_headers = dict({ (x, extract(x)) for x in headers }),
  # etc... 

[styurin]

If you have a genrule that creates a directory with several files, it's a bit of a pain to write another genrule rule for each (expected) output, just to be able to pass it to *_library (java, python, etc).

You can write a genrule that creates a zip file and pass it as an input in srcs: https://buckbuild.com/rule/java_library.html#srcs

[njlr]

Have any more plans been made regarding this?

Is rule[file] still the preferred syntax?