Dynamically set SyntaxNode prefix and line number

[Nhyland28]

Hi!

First off thanks for all the work that went into this library, it has exactly what I was looking for!

I'm creating a class which adds descriptions to dimensions when they do not exist. I'm using the advanced parsing method as I want to make sure all comments and formatting are kept. Here is the current iteration of my class (and the initial creation of the DocumentNode):

import lkml
from dataclasses import replace
from lkml.visitors import BasicTransformer
from lkml.tree import BlockNode, PairNode, SyntaxToken

with open(file_path, 'r') as file:
    tree = lkml.parse(file.read())

class DescriptionGeneratorTransformer(BasicTransformer):
    def visit_block(self, block: BlockNode) -> BlockNode:
        """Visit each block and adds a description node if one does not exist."""
        block = super().visit_block(block)
        if block.type in ['dimension', 'measure', 'dimension_group']:
            nodes = [node.type.value for node in block.container.items]
            if 'description' not in nodes:
                old_block = block.container.items
                old_prefix = block.container.items[0].value.prefix
                print(len(old_prefix))
                old_line_number = block.container.items[0].value.line_number
                print(old_line_number)
                new_node = PairNode(type=SyntaxToken('description', prefix='  '), value=SyntaxToken(f'"This is a description for the {block.name} dimension."', suffix='\n'))
                new_block = old_block + (new_node,)
                replacement_block = replace(block, container=replace(block.container, items=new_block))
                
                return replacement_block
            else:
                return block
        else:
            return block
            
new_tree = tree.accept(DescriptionGeneratorTransformer())

For the new description PairNode I'm trying to dynamically assign the type's SyntaxToken's prefix value so that it aligns with the other objects. My idea was that it could just copy the prefix value from the first PairNode in the same BlockNode. When I try to grab that value it returns an empty string (see the print(len(old_prefix)). My expectation is that it should return a string with 2 whitespaces . I could hardcode it (like I have it in the above class definition) but I would rather use that old_prefix value in case other LookML files are formatted differently.

Additionally, I would love to have the description become the first parameter in the dimension. Currently the above class places it last. I was thinking this is where I could use SyntaxToken's line_number value; however, I haven't been able to get it to work. My fear is that this would actually be a pretty complex change because I would potentially have to change the line_number for every SyntaxToken in the BlockNode (e.g. move them down one). Any thoughts or ideas on how I could make the description the first element?

Current:

  dimension: ascent_status {
    type: number
    sql: ${TABLE}."ASCENT_STATUS" ;;
    description: "This is a description for the ascent_status dimension."
}

Goal:

  dimension: ascent_status {
    description: "This is a description for the ascent_status dimension."
    type: number
    sql: ${TABLE}."ASCENT_STATUS" ;;
  }

[Nhyland28]

I actually think I figured out a way to make description the first key value pair in the dimension block and it's pretty simple! You can just switch the order of the concatenation that creates new_block.

Old version:

new_block = old_block + (new_node,)

Fixed (so description is first):

new_block = (new_node,) + old_block

Still trying to figure out a way to dynamically map the suffix for that node.

[joshtemple]

Hey @Nhyland28! Thanks for the detailed writeup. Not many people have used advanced parsing mode (to my knowledge) so totally possible you're encountering some rough edges of the library. I'm open to suggestions for improving it.

You figured out the correct way to make the description node the first item in the list by changing the order. That looks correct.

The way lkml assigns whitespace to each SyntaxToken probably could be improved, sometimes it feels a bit arbitrary. In this case, the whitespace you're looking for is actually in old_block.left_brace.suffix (the suffix of the { token rather than the prefix of the first PairNode). Does that give you what you need to fix the code?

Out of curiosity, how are you adding descriptions dynamically? Are you using ChatGPT or something to populate them?