Tool for analyzing code

[markfirmware]

I'd like a tool to help me as a code author to produce textual and visual commentary on the architecture and design principles and decisions used to write my code. Can I start with zig fmt to do that, or should I try something like f# or smalltalk?

For example, I am using stateful singletons to great effect in https://github.com/markfirmware/zig-bare-metal-microbit - I'd like something like a script that will find these in the code and list them or diagram them for the audience.

Working from https://github.com/markfirmware/zig-bare-metal-microbit/blob/master/mission03_model_railroad_button_controlled_pwm.zig

I can run this egrep and see that all the structs (except Button) are stateful singletons in that they have at least one var and no fields. Button has no var and at least one field so it is a record, maybe an object if it has functions (roughly speaking).

egrep '(struct|var .*= undefined;|:.*,$)' mission03_model_railroad_button_controlled_pwm.zig

const CycleActivity = struct {
    var cycle_counter: u32 = undefined;
    var cycle_time: u32 = undefined;
    var cycles_per_second: u32 = undefined;
    var last_cycle_counter: u32 = undefined;
    var last_cycle_start: ?u32 = undefined;
    var max_cycle_time: u32 = undefined;
    var up_time_seconds: u32 = undefined;
    var up_timer: TimeKeeper = undefined;
const ThrottleActivity = struct {
    var buttons: [2]Button = undefined;
    var pwm_loop_back_counter: u32 = undefined;
    const Button = struct {
        down_count: u32,
        index: u32,
        is_pressed: bool,
        is_simulation_pressed: bool,
        mask: u32,
        up_count: u32,
    const LedScroller = struct {
        var column: u32 = undefined;
        var displayed_percent: u32 = undefined;
        var text: []u8 = undefined;
        var text_buf: [text_len_max]u8 = undefined;
        var timer: TimeKeeper = undefined;
    const Throttle = struct {
        var pwm_width_ticks: u32 = undefined;
        var percent: u32 = undefined;
const TerminalActivity = struct {
    var keyboard_column: u32 = undefined;
    var prev_led_image: u32 = undefined;
    var prev_now: u32 = undefined;

I am particularly interested in something that helps me author commentary, although such a tool could be useful for studying someone else's product.

[LemonBoy]

The stdlib has a parse module that's also used by zig fmt, you can use that as a base for your tool

[markfirmware]

Thanks - this is working well.

pub fn main() !void {
    const allocator = &std.heap.FixedBufferAllocator.init(buf[0 ..]).allocator;
    const tree = try zig.parse(allocator, source);
    var i: u32 = 0;
    while (tree.root_node.iterate(i)) |trunk| : (i += 1) {
        findStruct(tree, 0, i, trunk);
    }
}

fn findStruct(tree: *Tree, depth: u32, n: u32, trunk: *Node) void {
    if (trunk.cast(VarDecl)) |var_decl| {
         if (trunk.iterate(0)) |container_node| {
             if (container_node.cast(ContainerDecl)) |container_decl| {
                  if (mem.eql(u8, tree.tokenSlice(container_decl.kind_token), "struct")) {
                      const struct_name = tree.tokenSlice(var_decl.name_token);
                      var has_field = false;
                      var has_state = false;
                      var i: u32 = 0;
                      while (container_node.iterate(i)) |node| : (i += 1) {
                          if (node.cast(VarDecl)) |state_decl| {
                              if (mem.eql(u8, tree.tokenSlice(state_decl.mut_token), "var")) {
                                  has_state = true;
                              }
                          } else if (node.cast(ContainerField)) |container_field| {
                              has_field = true;
                          }
                      }
                      if (has_state and !has_field) {
                          var d: u32 = 0;
                          while (d < depth) : (d += 1) {
                              warn("    ", .{});
                          }
                          warn("stateful singleton {}\n", .{struct_name});
                      }
                      i = 0;
                      while (container_node.iterate(i)) |node| : (i += 1) {
                          if (node.cast(VarDecl)) |state_decl| {
                              if (!mem.eql(u8, tree.tokenSlice(state_decl.mut_token), "var")) {
                                  findStruct(tree, depth + 1, i, node);
                              }
                          }
                      }
                  }
              }
         }
    } else {
        var i: u32 = 0;
        while (trunk.iterate(i)) |node| : (i += 1) {
            findStruct(tree, depth + 1, i, node);
        }
    }
}

const ast = zig.ast;
const ContainerDecl = ast.Node.ContainerDecl;
const ContainerField = ast.Node.ContainerField;
const mem = std.mem;
const Node = ast.Node;
const source = @embedFile("mission03_model_railroad_button_controlled_pwm.zig");
const std = @import("std");
const Tree = ast.Tree;
const VarDecl = ast.Node.VarDecl;
const warn = std.debug.warn;
const zig = std.zig;

var buf:[1024 * 1024] u8 = undefined;

[markfirmware]

@alexnask @andrewrk

./src/main.zig:6:20: error: no member named 'root_node' in struct 'std.zig.ast.Tree'
        while (tree.root_node.iterate(i)) |trunk| : (i += 1) {

Can you advise whether my little program is in the right architecture for updating, or should I start from scratch? For example, is this still close to how things now work?

                      if (node.cast(VarDecl)) |state_decl| {
                          if (mem.eql(u8, tree.tokenSlice(state_decl.mut_token), "var")) {

Regards,
Mark

[Luukdegram]

Hi @markfirmware,

Hopefully, my next comments can help you further.

./src/main.zig:6:20: error: no member named 'root_node' in struct 'std.zig.ast.Tree'
        while (tree.root_node.iterate(i)) |trunk| : (i += 1) {

The AST was restructured, and now contains node indices rather than pointers to a Node.
The root node has always index 0. So for example: tree.nodes.items(.tag)[0] == .root.

Can you advise whether my little program is in the right architecture for updating, or should I start from scratch? For example, is this still close to how things now work?

                      if (node.cast(VarDecl)) |state_decl| {
                          if (mem.eql(u8, tree.tokenSlice(state_decl.mut_token), "var")) {

Regards,
Mark

As the tree now contains indices and you do not operate on the Node directly, you will first have to check for the tag of a node using something like:

const node_tags = tree.nodes.items(.tag);

if (node_tags[some_node_index] == .global_var_decl) {}

Which brings me to the next part: A lot of the old tags have been split up in multiple tags to differentiate them, such as VarDecl became global_var_decl, local_var_decl, simple_var_decl and aligned_var_decl. And some other tags have been renamed/removed/added etc.

Ah good way to find out how the new structure works is by checking out https://github.com/ziglang/zig/blob/master/lib/std/zig/render.zig

You can also see what I've been changing in ZLS so far to update it to the latest master:
zigtools/zls#238

PS: Tree contains quite a fair bit of handy functions to help you get the correct nodes/tokens for specific structures. e.g. tree.fnProto(node); returns you ast.full.FnProto containing the function tokens, an iterator for parameters, return type node and more. Might be worth checking those out (you can see their usage in render.zig)

[markfirmware]

Thanks @Luukdegram ! I have found various answers in render.zig - is there a single function that will provide for any given node the entire lexeme for that node? I see renderTokenSlice which seems close.

[Luukdegram]

@markfirmware Unfortunately there's not. You'll have to run the nodes through a switch and handle them accordingly.

Edit: Seems like I misunderstood what you wanted. See ifreund's comment :)
PS: tokenSlice() operates on a token index, not on a node index.

[ifreund]

Thanks @Luukdegram ! I have found various answers in render.zig - is there a single function that will provide for any given node the entire lexeme for that node? I see renderTokenSlice which seems close.

Tree.tokenSlice() does this. The existence of renderTokenSlice() an implementation detail of how zig fmt handles multi line string literals.