Build system: Add support for custom exports

[ikskuh]

Right now, the build system allows passing two types of information between dependency edges:

• *std.Build.Step.Compile
• *std.Build.Module

I propose that we should add support to pass arbitrary types between dependency edges, so more
advanced projects like Mach or MicroZig can improve the user experience by passing around types of their build automation.

A potential implementation could look like this:

pub fn addCustomExport(b: *std.Build, comptime T: type, name: []const u8, value: T) *T;

pub fn customExport(dep: *std.Build.Dependency, comptime T: type, name: []const u8) *const;

Usage Example

In MicroZig, we have our own Target type that encodes relevant information for embedded targets including post processing steps, output format and so on.

It would be nice to have board support packages just expose MicroZig.Target instead of doing the hackery we have right now.

With this proposal, the user-facing build script could look like this:

const MicroZig = @import("microzig-build");

pub fn build(b: *std.Build) void {
  const microzig = MicroZig.Sdk.init(b, b.dependency("microzig", .{}));
  const raspberrypi_dep = b.dependency("microzig/bsp/raspberrypi", .{});
  
  // Access the custom target exported by our board support package:
  const pico_target = raspberrypi_dep.customExport(
    MicroZig.Target, 
    "board/raspberrypi/pico",
  );

  const firmware = microzig.add_firmware(.{
      .name = "blinky",
      .root_source_file = b.path("src/blinky.zig"),
      .target = pico_target,
      .optimize = optimize,
  });
  microzig.install_firmware(firmware);
}

While the board support package might look something like this:

// bsp/linux/build.zig
const MicroZig = @import("microzig-build");

pub fn build(b: *std.Build) void {
  const pico_sdk = b.dependency("pico-sdk", .{});

  const pico = b.addCustomExport(MicroZig.Target, "microzig/bsp/raspberrypi", .{
      .name = "RaspberryPi Pico",
      .vendor = "RaspberryPi",
      .cpu = MicroZig.cpus.cortex_m0,
  });
  pico.add_include_path(pico_sdk.path("src/rp2040/hardware_structs/include"));
}

Remarks

This feature most likely depends on the @typeId proposal: #19858

[rohlem]

I thought it was already possible to @import dependencies and fully use their build.zig contents as a Zig module (i.e. struct namespace type).

Why doesn't something like the following work in status-quo? What benefit does customExport bring?

//user
const mzb = @import("microzig-build");

const board_providers = mzb.default_providers; //could hold anything you need
const board_provider_config = ...; //could hold anything, even another @import
const result: mzb.Target = mzb.getTarget( //can freely return any type you need
  "board/raspberrypi/pico",
  board_providers,
  board_provider_config,
).?;

//microzig-build
pub const Target = struct{
  name: []const u8,
  vendor: []const u8,
  cpu: ...,
  dependencies: []const ...,
};
const board_providers = ...;
pub fn getTarget(name: []const u8, board_providers: anytype, board_provider_config: anytype) ?Target {
  return for (board_providers) |board_provider| {
    if (board_provider.getTarget(name, board_provider_config)) |t| break t;
  } else null;
}

//board_provider.zig
const mzb = @import("microzig-build");

pub fn getTarget(name: []const u8, board_provider_config: anytype) ?Target {
  _ = board_provider_config; //could be anything, even the type of another user-side @import
  if (@import("std").mem.eql(name, "board/raspberrypi/pico")) return .{
      .name = "RaspberryPi Pico",
      .vendor = "RaspberryPi",
      .cpu = mzb.cpus.cortex_m0,
      .dependencies = &.{
          .{ .name = "libusb", .module = libusb_mod },
      },
  };
  return null;
}

It would be nice to have board support packages just expose MicroZig.Target instead of doing the hackery we have right now.

Can you elaborate, roughly, on what sort of "hackery" is currently happening?
In my opinion having access to build.zig-s of dependencies via @import, and modeling whatever logic you need in userland, is a very flexible and valuable feature.
I don't see how passing values with string names via an std.Build-function is a cleaner mechanism than what I imagine already possible today.

I'm also not clear on which part of this would require a runtime-type-id concept - all dependencies are present at build time and compiled together AFAIU.

[ikskuh]

In my opinion having access to build.zig-s of dependencies via @import, and modeling whatever logic you need in userland, is a very flexible and valuable feature.

Except you cannot access the dependencies of that build.zig file at all. Dependencies will be only instantiated for the build.zig when you invoke it and call b.dependency(…).

Assume you want to pass a module inside a custom command that imports from another dependency. You can't model that right now, because there's no clean way to obtain a handle to the dependency tree

I'm also not clear on which part of this would require a runtime-type-id concept - all dependencies are present at build time and compiled together AFAIU.

How would you store a comptime T: type inside a std.Build structure and be able to later obtain the same type back? By name won't work because then you have type confusion real quick. Without typeId, it would require to make std.Build generic over the types you want to store via customExport

Can you elaborate, roughly, on what sort of "hackery" is currently happening?

We do pass a pointer via .dependency("...", .{ .context_ptr = @intFromPtr(&context) }) down the road so we can call methods on an instance of an object

[rohlem]

Dependencies will be only instantiated for the build.zig when you invoke it and call b.dependency(…).

I don't see why the requirement of calling b.dependency is limiting, or what customExport would change about it, so I assume I'm misunderstanding something.
EDIT: If you're talking about the dependencies of a dependency, then my follow-up question is how would you get the *std.Build.Dependency to that to call customExport on it?
If the middle man is the microzig-build module, then to me it seems the same way it can route a *std.Build.Dependency of that type it could also instead route the type representing its build.zig.

I assume that passing the user's build.zig struct via @This() to import-ed functions, say mzb.init(@This()),
so the init function can access all pub declarations in @This(), also isn't a way for you to share the required data?
(You could instruct users to provide a pub var mzb_context: mzb.Context = mzb.init(@This()); for your build framework to use.)

Assume you want to pass a module inside a custom command that imports from another dependency.

What do you mean by command in this context? A custom top-level-step invoked by zig build custom-step? A step at some other point in the build graph?
Or do you mean a compiled executable executed via a std.Build.Step.Run?

Can you elaborate, roughly, on what sort of "hackery" is currently happening?

We do pass a pointer via .dependency("...", .{ .context_ptr = @intFromPtr(&context) }) down the road so we can call methods on an instance of an object

I'm confused whether @intFromPtr is necessary here - type-erasing the pointer type by @ptrCast between *anyopaque and *ActualImplementation should do the same, right?
(Both sides already need to privately know the ActualImplementation type to use the value in a meaningful way.)

[ikskuh]

Some more elaboration:

The follow use case isn't possible with the proposed solution of "just using @import":

Library Code

build.zig

const MicroZig = @import("microzig-build");

pub fn build(b: *std.Build) void {
  const pico_sdk = b.dependency("pico-sdk", .{});

  const pico = b.addCustomExport(MicroZig.Target, "microzig/bsp/raspberrypi", .{
      .name = "RaspberryPi Pico",
      .vendor = "RaspberryPi",
      .cpu = MicroZig.cpus.cortex_m0,
  });
  pico.add_include_path(pico_sdk.path("src/rp2040/hardware_structs/include"));
}

build.zig.zon

.{
    …,
    .dependencies = .{
        .@"pico-sdk" = .{ … }
    },
}

There's no way of getting a handle of the std.Build instance required for .dependency("pico-sdk", .{}) without invoking .dependency("microzig/bsp/raspberrypi") on the BSP itself, so you have to do that anyways.

After that, you can then pass the *std.Build.Dependency.builder down to your custom function so that can then access it as if you would've called fn build(b: *std.Build) void instead.

And now we're at a point where we could've just used that function in the first place.

Also @lazyImport/lazyDependency won't work with that approach which is sad

I'm confused whether @intFromPtr is necessary here - type-erasing the pointer type by @ptrCast between *anyopaque and *ActualImplementation should do the same, right?

You cannot pass any pointers through the command line interface of zig build, so you can't pass them down into the string serializer of .dependency either, supported types are here:
https://ziglang.org/documentation/0.12.0/std/#std.Build.userInputOptionsFromArgs

What do you mean by command in this context? A custom top-level-step invoked by zig build custom-step? A step at some other point in the build graph?

Sorry, i just meant an exported function form a dependency build.zig

(You could instruct users to provide a pub var mzb_context: mzb.Context = mzb.init(@this()); for your build framework to use.)

That won't work as the configuration may be run concurrently and you cannot have the same dependency invoked with different parameters, you have to go through a *std.Build instance somehow

[castholm]

I've read the issue description and the discussion a few times over and I'm not sure I understand exactly what is being asked for, but I believe the functionality you want can already cleanly implemented in user space today using @import/b.lazyImport. Simple reduced example:

./
├── main/
│   ├── build.zig
│   └── build.zig.zon
├── microzig/
│   ├── build.zig
│   └── build.zig.zon
├── raspberrypi/
│   ├── build.zig
│   └── build.zig.zon
└── pico-sdk/
    └── foo/
        └── bar/
            └── include/
                ├── a.txt
                └── b.txt

// main/build.zig.zon
.{
    .name = "main",
    .version = "0.0.0",
    .dependencies = .{
        .microzig = .{
            .path = "../microzig",
        },
        .raspberrypi = .{
            .path = "../raspberrypi",
        },
    },
    .paths = .{""},
}

// main/build.zig
const std = @import("std");
const microzig = @import("microzig");
const raspberrypi = @import("raspberrypi");

pub fn build(b: *std.Build) void {
    const rp_pico_target: microzig.Target = raspberrypi.getTarget(b, "pico").?;
    std.debug.print("{s}\n", .{rp_pico_target.name});

    const rp_400_target: microzig.Target = raspberrypi.getTarget(b, "400").?;
    std.debug.print("{s}\n", .{rp_400_target.name});

    b.installDirectory(.{
        .source_dir = rp_pico_target.include_path.?,
        .install_dir = .prefix,
        .install_subdir = "",
    });
}

// microzig/build.zig.zon
.{
    .name = "microzig",
    .version = "0.0.0",
    .paths = .{""},
}

// microzig/build.zig
const std = @import("std");

pub fn build(b: *std.Build) void {
    _ = b;
}

pub const Target = struct {
    name: []const u8,
    include_path: ?std.Build.LazyPath = null,
};

// raspberrypi/build.zig.zon
.{
    .name = "raspberrypi",
    .version = "0.0.0",
    .dependencies = .{
        .microzig = .{
            .path = "../microzig",
        },
        .@"pico-sdk" = .{
            .path = "../pico-sdk",
        },
    },
    .paths = .{""},
}

// raspberrypi/build.zig
const std = @import("std");
const microzig = @import("microzig");

pub fn build(b: *std.Build) void {
    _ = b;
}

pub fn getTarget(b: *std.Build, name: []const u8) ?microzig.Target {
    if (std.mem.eql(u8, name, "pico")) {
        const this_dep = b.dependencyFromBuildZig(@This(), .{});
        const pico_sdk_dep = this_dep.builder.dependency("pico-sdk", .{});
        const include_path = pico_sdk_dep.path("foo/bar/include");
        return .{ .name = "RaspberryPi Pico", .include_path = include_path };
    }
    if (std.mem.eql(u8, name, "400")) {
        return .{ .name = "RaspberryPi 400" };
    }
    return null;
}

Here, the main root package calls a function exported by raspberrypi, which in turn references a dependency-relative path in pico-sdk (b.dependencyFromBuildZig(@This(), .{}) is the key here) and packages it up as a microzig.Target. For demonstration purposes it installs some dummy files from pico-sdk. This builds and runs perfectly fine without issue.

Obviously it is a bit more involved if you also want to pass along targets and build options, but that's mainly a question of designing your exported APIs in a clever and intuitive way and not something the build system itself will restrict you from doing.

[ikskuh]

That's indeed an interesting solution. I didn't know about dependencyFromBuildZig and that might actually solve the problem okayish.

I still think it's a hack and the passing of custom types is definitly a valid use case for the build system, as it would streamline and simplify a lot of stuff

[pfgithub]

Here's a hacky way to do custom exports currently:

const AnyPtr = struct {
    id: [*]const u8,
    val: *const anyopaque,
};
fn exposeArbitrary(b: *std.Build, name: []const u8, comptime ty: type, val: *const ty) void {
    const valv = b.allocator.create(AnyPtr) catch @panic("oom");
    valv.* = .{
        .id = @typeName(ty),
        .val = val,
    };
    const name_fmt = b.fmt("__exposearbitrary_{s}", .{name});
    const mod = b.addModule(name_fmt, .{});
    // HACKHACKHACK
    mod.* = undefined;
    mod.owner = @ptrCast(@alignCast(@constCast(valv)));
}
fn findArbitrary(dep: *std.Build.Dependency, comptime ty: type, name: []const u8) *const ty {
    const name_fmt = dep.builder.fmt("__exposearbitrary_{s}", .{name});
    const modv = dep.module(name_fmt);
    // HACKHACKHACK
    const anyptr: *const AnyPtr = @ptrCast(@alignCast(modv.owner));
    std.debug.assert(anyptr.id == @typeName(ty));
    return @ptrCast(@alignCast(anyptr.val));
}

[dasimmet]

I think what it comes down to at the moment is wrapping functions that provide custom functionality to expose them and have an internal implementation function, since b.dependencyFromBuildZig(@This(), args) does not work from the root std.Build as it does not contain its own dependency:

fn build(b: *std.Build) void {
     const custom_type = CustomBuildType.initInternal( b, .{});
    // do something with the CustombuildType
}

pub const CustomBuildType = struct{
    pub const Options = struct{};


    // This can be used from an @import() of this dependency
    pub fn init(b: *std.Build, opt: Options, args: anytype) CustomBuildType {
        return initInternal( b.dependencyFromBuildZig(@This(), args).builder, opt);
    }
    
    // b is "guaranteed" to be the current std.Build in private scope
    fn initInternal(b: *std.Build, opt: Options) CustomBuildType {
        _ = b;
        _ = opt;
        return .{};
    }
};