Skip to content

Add Support for Batch-Normalization to Streaming Log Outputs #59

@dchoi-viant

Description

@dchoi-viant

Related to mixed non-batch and batch usage from Issue #58.

We see that https://github.com/viant/mly/blob/v0.19.x/service/stream/service.go#L91 service.stream.(*Service).Log() sends logs with the input directly copied into the logged entry.

This causes some issues for a use case of using the logged outputs to feed into a GBQ (or any structured, list-capable schema) table - since the non-batched vs batched inputs have differing types (scalar vs. list, respectively), the same log entry won't work without some form of manipulation.

The main question is if this should be an option enabled on the streaming or if this is strictly a concern for our use case to address.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions