Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CT-1549] Reorganize structure of logging events to have two top level keys #6311

Closed
gshank opened this issue Nov 23, 2022 · 0 comments · Fixed by #6553
Closed

[CT-1549] Reorganize structure of logging events to have two top level keys #6311

gshank opened this issue Nov 23, 2022 · 0 comments · Fixed by #6553
Assignees
Labels
Milestone

Comments

@gshank
Copy link
Contributor

gshank commented Nov 23, 2022

In a previous effort we implemented a different structure for our logging events, using a protobuf "oneof" for all of the individual sets of structured data. We had to abandon that approach because of very poor performance in Python.

Currently the logging events are structured so that there is a top-level "info" field which contains all of the common fields in each event, such as "msg", "level", "thread", "extra", "category", etc. In addition there are other top-level fields for each piece of structured data in the specific logging event.

After looking at other possible event organizations, we are thinking of switching to a structure with only two top-level fields, "info" and "data", where "data" would have the specific event's structured information.

event2 = {           
    "info": {       
        "name": "LogTestResult",                            
        "code": "Q007",
        "msg": ".......",  
        "level": "info",                 
        "invocation_id": "5ef0dcdb-5370-48f7-b130-8163c5a9a448",
        "pid": 12647,                         
        "thread": "MainThread",   
        "ts": "2022-11-17T18:14:27.578648Z",
        "extra": {},     
        "category": "",
    },                                                   
    "data": {                     
        "name": "my_test",
        "status": "pass",
        "index": 1,                         
        "num_models": 4,
        "execution_time": 0.23456,
        "num_failures": 1,
        "node_info": {
            "node_path": "marts/people.sql", 
            "node_name": "people", 
            "unique_id": "model.grs_test.people", 
            "resource_type": "model", 
            "materialized": "table", 
            "node_status": 
            "started", 
            "node_started_at": "2022-11-17T18:14:28.375976", 
            "node_finished_at": null
        },
    },
}

Implementing this would require us to have two messages for every logging event, possibly a SomeEventDetails and a SomeEvent, where SomeEvent would contain an EventInfo message and the SomeEventDetails message. Or alternatively leave the current names the same and make the containing message something like SomeEventMsg, containing an EventInfo and a SomeEvent object. We will also have to refactor the code in "fire_event' to create the containing message, the EventInfo message, and create the final SomeEvent object.

@github-actions github-actions bot changed the title Reorganize structure of logging events to have two top level keys [CT-1549] Reorganize structure of logging events to have two top level keys Nov 23, 2022
@jtcohen6 jtcohen6 added this to the v1.4 milestone Jan 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants