Add WalkMetas* functions and meta.Name field #1089

joelanford · 2023-04-22T20:15:52Z

Description of the change:
These function are like WalkFS except that callers provide a function that handles a Meta object at a time, rather than an entire file's worth of FBC objects.

This PR also adds Name string to the Meta struct, which is necessary information to distinguish meta objects that have the same schema and package (e.g. olm.channel and olm.bundle objects)

Motivation for the change:
For callers that can operate on a single meta at a time, these functions will provide significant reductions in memory usage. The catalogd project will likely make use of this function.

Closes #1080

Reviewer Checklist

Implementation matches the proposed design, or proposal is updated to match implementation
Sufficient unit test coverage
Sufficient end-to-end test coverage
Docs updated or added to /docs
Commit messages sensible and descriptive

codecov · 2023-04-22T20:21:59Z

Codecov Report

Merging #1089 (f33af22) into master (0aebc3c) will increase coverage by 0.03%.
The diff coverage is 73.61%.

❗ Current head f33af22 differs from pull request most recent head 7e4f23a. Consider uploading reports for the commit 7e4f23a to get more accurate results

@@            Coverage Diff             @@
##           master    #1089      +/-   ##
==========================================
+ Coverage   52.60%   52.64%   +0.03%     
==========================================
  Files         107      107              
  Lines        9389     9411      +22     
==========================================
+ Hits         4939     4954      +15     
- Misses       3523     3526       +3     
- Partials      927      931       +4

Impacted Files	Coverage Δ
alpha/declcfg/declcfg.go	`80.48% <73.33%> (-19.52%)`	⬇️
alpha/declcfg/load.go	`75.00% <73.80%> (-1.53%)`	⬇️

... and 1 file with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

joelanford · 2023-04-24T15:23:58Z

/hold
I want a few reviews on this one, so I don't want a single lgtm allowing this to merge.

anik120

Overall looks good to me.

The catalogd project will likely make use of this function.

I was a little mislead by this. If I'm not wrong, nothing changes in terms of wirings for clients, I.e clients still call LoadReader like they have been. So it's more The catalogd project will likely benefit from this function. Is that correct?

anik120 · 2023-04-24T16:13:42Z

alpha/declcfg/load.go

+			}
+			return walkFn(nil, err)
+		}
+		doc = []byte(strings.NewReplacer(`\u003c`, "<", `\u003e`, ">", `\u0026`, "&").Replace(string(doc)))


Could we have some documentation for what the values for the NewReplacer method are? I.e what exactly are we achieving with this line?

See some context here: #1089 (comment)

everettraven

Looks good overall, just have a few nits/comments - I don't think any of them should block this PR from merging though.

alpha/declcfg/load.go

stevekuznetsov · 2023-04-24T16:46:57Z

alpha/declcfg/load.go

+			}
+			return walkFn(nil, err)
+		}
+		doc = []byte(strings.NewReplacer(`\u003c`, "<", `\u003e`, ">", `\u0026`, "&").Replace(string(doc)))


Isn't there a bytes replacer that keeps you from having to double-allocate memory here?

I don't see a bytes.NewReplacer. There is bytes.Replace(All), but as far as I can tell, they only handle one substitution at a time. So I think options are:

Roll with this and tolerate the double-allocation

Call bytes.Replace in a loop for each replacement. (seems worse from an allocation standpoint since we have 3 replacements)

Roll our own bytes replacer.

Figure out a way to not require this replacement in the first place. It's been awhile, but iirc, its because an encoder is used somewhere along the line that we don't control and therefore can't set enc.SetEscapeHTML(false)

Not bother with the replacement and let the unicode show up in the output.

I mis-remembered this, sorry - I used this library in the past and found it nice.

One producer of these artifacts is opm render itself. :) If we output YAML, it escapes html-hostile runes like these, so we have to replace them on ingest.

I would argue that the component trying to do "user-facing YAML" should choose how to encode whatever it's outputting and call it good. Having to incur this burden in the entire back-end stack is a bit wild.

alpha/declcfg/load.go

alpha/declcfg/load_test.go

alpha/declcfg/declcfg.go

joelanford · 2023-04-24T20:43:00Z

clients still call LoadReader like they have been

@anik120 That section of code reads the entire contents of the catalog into memory (twice actually: once into a bytes.Buffer, and a second time into declcfg.DeclarativeConfig), returns it from the function, and then the caller calls createPackage and createBundleMetadata, each of which basically iterate the appropriate schema-based lists.

It would be way more performant if we did declcfg.LoadMetasReader(podLogs, walkFunc), where walk func could:

directly create BundleMetadata when it sees an olm.bundle
collect packages and channels up to eventually be used to create Package objects.

Later, if/when we treat metas a bit more opaquely, we can just sync every object directly to the underlying storage without having to collect anything up at all. (e.g. something closer to this, which was the original intent behind this PR in the first place.

These function are like WalkFS except that callers provide a function that handles a Meta object at a time, rather than an entire file's worth of FBC objects. For callers that can operate on a single meta at a time, these functions will provide significant reductions in memory usage. Signed-off-by: Joe Lanford <joe.lanford@gmail.com>

Signed-off-by: Joe Lanford <joe.lanford@gmail.com>

joelanford · 2023-04-25T03:12:43Z

alpha/declcfg/declcfg.go

@@ -93,17 +100,68 @@ func (m Meta) MarshalJSON() ([]byte, error) {
 }

 func (m *Meta) UnmarshalJSON(blob []byte) error {
+	blob = bytereplacer.New(`\u003c`, "<", `\u003e`, ">", `\u0026`, "&").Replace(blob)


I moved the bytes replacement here, which meant I could drop that extra unmarshal call into doc json.RawMessage when we're deciding the reader. Between the allocation improvement of a direct bytes replacer and the removal of the other unmarshal, we shave off ~7% of the runtime of opm render when I run it against the operatorhub catalog.

Not too shabby!

In a follow-up, I might take a look at an alternate mode of action.Render (maybe a new function alongside Run()?), which lets the caller use a walk function. That would save a bunch on memory and might shave off more time as well.

joelanford · 2023-04-25T03:13:49Z

alpha/declcfg/declcfg.go

+		// TODO: return an error that includes the the full JSON message,
+		//    the offset of the error, and the error message. Let callers
+		//    decide how to format it.
+		return errors.New(resolveUnmarshalErr(blob, err))


I moved this here because it was the only place access to the full json message existed after I removed the extra unmarshal to json.RawMessage.

everettraven

LGTM

openshift-ci · 2023-04-25T14:00:22Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: everettraven, joelanford

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [joelanford]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

joelanford · 2023-04-25T20:23:45Z

/hold cancel
I think this is ready to merge (just needs /lgtm)

For follow-ups, we have:

Refactor the unmarshal error formatting to return an error struct to give callers more formatting flexibility.
Look at improving WalkMetas* tests to assert on the contents of the objects.

everettraven · 2023-05-05T23:42:56Z

/LGTM

* add WalkMetasFS and WalkMetasReader functions These function are like WalkFS except that callers provide a function that handles a Meta object at a time, rather than an entire file's worth of FBC objects. For callers that can operate on a single meta at a time, these functions will provide significant reductions in memory usage. Signed-off-by: Joe Lanford <joe.lanford@gmail.com> * address PR feedback Signed-off-by: Joe Lanford <joe.lanford@gmail.com> --------- Signed-off-by: Joe Lanford <joe.lanford@gmail.com>

openshift-ci bot requested review from dinhxuanvu and everettraven April 22, 2023 20:16

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 22, 2023

joelanford force-pushed the walk-metas branch from eec8100 to 4c6dc80 Compare April 22, 2023 20:29

joelanford mentioned this pull request Apr 22, 2023

FBC updates operator-framework/olm-docs#290

Merged

joelanford changed the title ~~Add WalkMetasFS function~~ Add WalkMetas* functions and meta.Name field Apr 23, 2023

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 24, 2023

joelanford force-pushed the walk-metas branch from 4c6dc80 to 5bb1b5a Compare April 24, 2023 15:38

oceanc80 mentioned this pull request Apr 24, 2023

Optimize memory usage of unpacking FBC from pod logs operator-framework/catalogd#22

Closed

dinhxuanvu removed their request for review April 24, 2023 15:40

anik120 reviewed Apr 24, 2023

View reviewed changes

everettraven approved these changes Apr 24, 2023

View reviewed changes

alpha/declcfg/load.go Show resolved Hide resolved

alpha/declcfg/load.go Outdated Show resolved Hide resolved

alpha/declcfg/load.go Show resolved Hide resolved

stevekuznetsov reviewed Apr 24, 2023

View reviewed changes

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 25, 2023

joelanford force-pushed the walk-metas branch from d71b0d6 to ad98a2c Compare April 25, 2023 02:50

openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 25, 2023

address PR feedback

7e4f23a

Signed-off-by: Joe Lanford <joe.lanford@gmail.com>

joelanford force-pushed the walk-metas branch from ad98a2c to 7e4f23a Compare April 25, 2023 02:55

joelanford commented Apr 25, 2023

View reviewed changes

everettraven approved these changes Apr 25, 2023

View reviewed changes

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 25, 2023

openshift-ci bot assigned everettraven May 5, 2023

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label May 5, 2023

openshift-merge-robot merged commit e1eebae into operator-framework:master May 5, 2023
10 checks passed

This was referenced May 6, 2023

Follow-up: improve WalkMetas* tests #1092

Open

Follow-up: refactor FBC unmarshal error to struct #1093

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add WalkMetas* functions and meta.Name field #1089

Add WalkMetas* functions and meta.Name field #1089

joelanford commented Apr 22, 2023 •

edited

Loading

codecov bot commented Apr 22, 2023 •

edited

Loading

joelanford commented Apr 24, 2023

anik120 left a comment

anik120 Apr 24, 2023

joelanford Apr 24, 2023

everettraven left a comment

stevekuznetsov Apr 24, 2023

joelanford Apr 24, 2023

stevekuznetsov Apr 24, 2023

grokspawn May 15, 2023

stevekuznetsov May 15, 2023

joelanford commented Apr 24, 2023 •

edited

Loading

joelanford Apr 25, 2023 •

edited

Loading

joelanford Apr 25, 2023

everettraven left a comment

openshift-ci bot commented Apr 25, 2023

joelanford commented Apr 25, 2023

everettraven commented May 5, 2023

Add WalkMetas* functions and meta.Name field #1089

Add WalkMetas* functions and meta.Name field #1089

Conversation

joelanford commented Apr 22, 2023 • edited Loading

codecov bot commented Apr 22, 2023 • edited Loading

Codecov Report

joelanford commented Apr 24, 2023

anik120 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

everettraven left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joelanford commented Apr 24, 2023 • edited Loading

joelanford Apr 25, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

everettraven left a comment

Choose a reason for hiding this comment

openshift-ci bot commented Apr 25, 2023

joelanford commented Apr 25, 2023

everettraven commented May 5, 2023

joelanford commented Apr 22, 2023 •

edited

Loading

codecov bot commented Apr 22, 2023 •

edited

Loading

joelanford commented Apr 24, 2023 •

edited

Loading

joelanford Apr 25, 2023 •

edited

Loading