Skip to content

Pipeline support wip#3

Merged
mebegu merged 68 commits into
mainfrom
pipeline-support-wip
May 12, 2025
Merged

Pipeline support wip#3
mebegu merged 68 commits into
mainfrom
pipeline-support-wip

Conversation

@mebegu
Copy link
Copy Markdown

@mebegu mebegu commented Apr 28, 2025

No description provided.

@ddelnano
WIP for FileSource connector
5a920fa 
Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
Working FileSource connector with occassional segfaults from race con…
5438415 
…dition with table store schema creation

Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
Add pxlog module, FileSource function and the associated tests
1023689 
Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
Add file source mutation support to query broker
7606f37 
Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
Add file source controller to metadata service
8f2ae0e 
Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
Integrate file source controller into metadata service
9d62c51 
Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
Register metadata service's FileSource RPC methods. Add missing mutat…
08356f4 
…ion info support to query broker

Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
Add GetFileSourceStatus UDTF
b4da693 
Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
Remove debug message
0b2c17c 
Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
Ensure FileSourceManager is passed through to avoid segfault
d10362b 
Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
Add pxl support for deleting file sources
c6ff273 
Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
Fix mutation executor bug that prevented UI from showing table result…
6f73901 
…s (stuck on "Prepare schema")

Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
FileSourceConnector and PEM changes to get the system working
4c73e71 
Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
Update go protos
2c91ee2 
Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
Linting changes
c9f452a 
Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
Move Table::Cursor to a non nested class and introduce base and sub c…
7269101 
…lasses for HotOnlyTable

Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
Add bare bones HotOnlyTable impl
62930a6 
Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
WIP implementing HotOnlyTable
af28ec3 
Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
Fix StoreWithRowAccounting.PopFront bug (returning unalloc'ed memory)…
64293fc 
…. Finish HotOnlyTable tests

Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
Update standalone PEM's file source manager to work with HotOnlyTable
d9528c9 
Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
Fix use after free issue detected by ASAN
0bfb559 
Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
Determine accurate file position by clearing error state (eof)
4b0552b 
Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
Use vector auto generated file in standalone pem client script
c434c9b 
Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
Add stirling file source connector e2e test
22970aa 
Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
Linting changes
c2c8259 
Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
Refactor file parsing to make it easier to support other formats
903a64a 
Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
Add stream_status table to track throughput of file sources
4565751 
Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
Add px-log-generator for use with Pixie log source
0bf70cf 
Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
Add initial version of sink_results table within OTelExportSinkNode
e5ab803 
Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
Add mutation id to Relation data types and make accessible from compi…
d9a90b6 
…ler state. Will be later renamed within a relation metadata field

Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
ddelnano and others added 19 commits February 28, 2025 20:07
@ddelnano
Remove extraneous logs
a5afbee 
Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
Revert unnecssary changes and add extra information to analyze log
8314380 
Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
Handle unstructured files in file source connector
a77bb9d 
Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
Implement exec node support for OTel log export
ac0d014 
Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
Add log support to OTelExportSinkIR
9d5e616 
Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
Fix misspelling
adc80bd 
Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
Add px.otel.log pxl module
9ec76ad 
Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
Ensure query broker CompileMutationRequests have otel and plugin conf…
444b762 
…iguration population

Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
Update pipeline status to support bpftrace
5a365ab 
Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
Enhance FileSourceConnector to support unstructured files
04cf9a6 
Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
Add tracepoint id string columen to GetTracepointStatus UDTF
dd05ec8 
Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
Fix bug in HotOnlyTable store
310eb74 
Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
Convert array and object JSON data values to strings so nested fields…
e001302 
… can be handled with native PxL

Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@mebegu
build prep for local dev
601ce9f
@mebegu
make file source persistent table
8ca941b
@ddelnano
Do not truncate string columns for file sources
958f5b6 
Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@mebegu
Merge branch 'ddelnano/pipeline-support-wip' into pipeline-support-wip
bb26cb6
@ddelnano
Add uuid column to file source data tables
0d1c0e0 
Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@mebegu
Merge branch 'ddelnano/pipeline-support-wip' into pipeline-support-wip
9fad0d3
@mebegu mebegu force-pushed the pipeline-support-wip branch from 35bb7a0 to 9fad0d3 Compare May 8, 2025 13:34
@mebegu mebegu marked this pull request as ready for review May 9, 2025 12:10
entlein added 2 commits May 9, 2025 14:51
@entlein
Update .bazelrc
ef5e22e 
testing my pr workflow against the new leading branch

Signed-off-by: Duck <70207455+entlein@users.noreply.github.com>
@entlein
Merge pull request #5 from k8sstormcenter/entlein-patch-2
5f7e4e7 
Update .bazelrc
@entlein entlein temporarily deployed to pr-actions-approval May 9, 2025 12:52 — with GitHub Actions Inactive
@mebegu mebegu changed the base branch from main to pipeline-support-wip-merged May 12, 2025 11:13
@mebegu mebegu changed the base branch from pipeline-support-wip-merged to main May 12, 2025 11:13
@mebegu mebegu merged commit e09ab3e into main May 12, 2025
35 of 56 checks passed
ddelnano pushed a commit that referenced this pull request Feb 25, 2026
@mebegu
Merge pull request #3 from k8sstormcenter/pipeline-support-wip
6c670cf 
Pipeline support wip
ddelnano pushed a commit that referenced this pull request Feb 25, 2026
@mebegu
Merge pull request #3 from k8sstormcenter/pipeline-support-wip
7cd1564 
Pipeline support wip
entlein pushed a commit that referenced this pull request Jun 2, 2026
@mebegu
Merge pull request #3 from k8sstormcenter/pipeline-support-wip
0d3d1b1 
Pipeline support wip
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@mebegu @entlein @ddelnano