KHI v0.56.0
.khi file format structure change
Previously .khi file format used JSON for storing its metadata and gzipped raw array buffer for long text part. This limits the maximum size of the logs due to the max string length accepted on V8.
This version changed .khi to use protocol buffer based file schema. It allows performant serialization/deserialization while we maintain file schema interface across backend and frontend.
Protocol buffer has limitation on the max size per single message, thus KHI concatenate multiple protos on simple KHI's container file format.
Comparison by file format update
| scenario | log volume | old format size | new format size | reduction ratio |
|---|---|---|---|---|
| basic scenario | 223464 logs | 128.2MB | 19.4MB | x 6.6 |
| larger scenario | 1386843 logs | 885.3MB | 144.4MB | x6.1 |
This is not only for the file size issue. KHI needs decompress string data from the file at the beginning of the load to perform filtering and consume a lot of memory(The larger scenario data with old format couldn't be loaded on my environment.) This allows frontend to open larger log data.
Note that even if we support larger log volume, still it's recommended to set appropriate filter to query. Larger logs takes longer time to filter and the inspection experience would be worse.
UI changes
More types of timelines
KHI was initially designed only for visualizing Kubernetes logs. Thus timeline layer count was fixed and it always has the structure APIVersion -> Kind -> Namespace -> Resource -> SubResource.
However, now we have a lot of types of visualization are placed with hierarchical relationships such as .status.conditions or .metadata.ownerReference. We recently added support for CSM TD resources or Managed airflow. These were needed to be placed on timeline with mapping their concept to kubernetes concept forcibly.
New change allows defining new timeline type for each parsers and the count of layers of timeline hierarchy is not limited.
With this change, Kubernetes resource hierarchy was changed to cluster name -> api version -> kind -> namespace -> resource -> subresource with aiming to support multi clusters.
New timeline filters
By the change of timeline types, KHI no longer guarantee given data contains namespace or kind timeline. It needed more widely applicable filtering mechanism.
CEL based filtering
The new filtering system is based on CEL based highly flexible filtering system. Now advanced user can input the filter directly. One of the example of this usage is to filter timelines by its manifest property.
The screenshot captures filtering timelines which has status.qosClass = Burstable. This allows us to find resources with more complex conditions.
Severity filtering
KHI hasn't supported severity based filtering. This was not from the architectural limit, but more like my own ideological policy. But now I supported it as there were many requests on it.
But note that severities in distributed system sometimes become misleading because each component decides its severity from its own perspective. Severity is not a universal criteria of logs.
Log body search and diff body search
Our log or diff components hasn't provided searching capability. It lead user to use in-browser search but it sometimes hit other elements not on the logs or the manifest.
Graph view integration on the dockable window
hevc.mov
Now KHI's graph view is integrated on the dockable window. It allows us to see the diagram along with the timeline.
What's Changed
Bug Fixes 🐛
- fix composer parsers to use GCPMainMessageFieldSet because log.MainMessageFieldSet is no longer used by default by @kyasbal in #792
- fixed CSM access log timeline style by @kyasbal in #796
- fix style override didn't reflect its status to preview by @kyasbal in #794
CI/CD ⚙️
- fix build script failure for releasing by @kyasbal in #790
- Simplified and optimized release workflows running on Cloud Build by @kyasbal in #797
Documentation 📝
- docs: fix typo in development guide (automativally → automatically) by @manumishra12 in #799
Other Changes 🛠
- Merge epic/file-schema-v6 into main by @kyasbal in #787
- improve cloud composer timeline styles by @kyasbal in #795
- docs: fix broken relative links in contributor/concept guides by @manumishra12 in #793
- Added timeline exclusion filter by @kyasbal in #800
- Added release note view by @kyasbal in #801
New Contributors
- @manumishra12 made their first contribution in #793
Full Changelog: v0.55.2...v0.56.0
