Skip to content

fix: SpecReader to escape embedded JSON files#733

Closed
disq wants to merge 6 commits intocloudquery:mainfrom
disq:fix/specreader-escape-json-embeds
Closed

fix: SpecReader to escape embedded JSON files#733
disq wants to merge 6 commits intocloudquery:mainfrom
disq:fix/specreader-escape-json-embeds

Conversation

@disq
Copy link
Copy Markdown
Member

@disq disq commented Mar 15, 2023

Should fix #9094

@disq disq requested a review from bbernays March 15, 2023 11:04
@disq disq requested a review from yevgenypats as a code owner March 15, 2023 11:04
@github-actions github-actions Bot added the fix label Mar 15, 2023
@codecov
Copy link
Copy Markdown

codecov Bot commented Mar 15, 2023

Codecov Report

Patch coverage: 100.00% and project coverage change: +0.07 🎉

Comparison is base (a0a58c4) 48.82% compared to head (50e86c1) 48.89%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #733      +/-   ##
==========================================
+ Coverage   48.82%   48.89%   +0.07%     
==========================================
  Files          70       70              
  Lines        6923     6933      +10     
==========================================
+ Hits         3380     3390      +10     
  Misses       3077     3077              
  Partials      466      466              
Impacted Files Coverage Δ
specs/spec_reader.go 71.34% <100.00%> (+1.70%) ⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Mar 15, 2023

⏱️ Benchmark results

Comparing with c9f95a5

  • DefaultConcurrencyDFS-2 resources/s: 10,962 ⬆️ 4.14% increase vs. c9f95a5
  • DefaultConcurrencyRoundRobin-2 resources/s: 11,902 ⬇️ 4.89% decrease vs. c9f95a5
  • Glob-2 ns/op: 195.6 ⬇️ 27.04% decrease vs. c9f95a5
  • TablesWithChildrenDFS-2 resources/s: 29,222 ⬆️ 8.80% increase vs. c9f95a5
  • TablesWithChildrenRoundRobin-2 resources/s: 28,321 ⬆️ 12.94% increase vs. c9f95a5
  • TablesWithRateLimitingDFS-2 resources/s: 28.46 ⬆️ 0.07% increase vs. c9f95a5
  • TablesWithRateLimitingRoundRobin-2 resources/s: 828.3 ⬇️ 4.90% decrease vs. c9f95a5
  • BufferedScanner-2 ns/op: 9.375 ⬇️ 25.23% decrease vs. c9f95a5
  • LogReader-2 ns/op: 30.75 ⬇️ 25.14% decrease vs. c9f95a5

Comment thread specs/testdata/creds2.json Outdated
@@ -0,0 +1 @@
{"key": "foo", "secret": "bar"} No newline at end of file
Copy link
Copy Markdown
Contributor

@hermanschaaf hermanschaaf Mar 15, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's test a JSON file with multiple lines as well (or only a file with multiple lines - if that works a file with a single line should also work)

Comment thread specs/spec_reader.go Outdated
encoder.SetEscapeHTML(false)
expandErr = encoder.Encode(string(content))
b := buffer.Bytes()
if l := len(b); l > 0 && b[l-1] == '\n' {
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

encoder.Encode always adds the trailing \n, there's no way to unset EscapeHTML otherwise.

Comment thread specs/spec_reader_test.go
t.Fatal(err)
}
if runtime.GOOS == "windows" {
expectedCfg = bytes.ReplaceAll(expectedCfg, []byte(`\n`), []byte(`\r\n`))
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are escaped sequences and not actual newline characters, so 2 and 4 bytes in length respectively.

Comment thread specs/spec_reader.go Outdated
Copy link
Copy Markdown
Contributor

@hermanschaaf hermanschaaf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is quite horrible 😅 , but also creative, and it solves a problem for our users. I still think we need to ban the use of file variables to expand yaml (see discussion in https://github.com/cloudquery/plugin-sdk/issues/480), but until we do that this should do 👍

@disq disq requested a review from candiduslynx March 15, 2023 16:54
disq and others added 2 commits March 16, 2023 09:26
Co-authored-by: candiduslynx <candiduslynx@users.noreply.github.com>
@disq
Copy link
Copy Markdown
Member Author

disq commented May 30, 2023

302 moved: cloudquery/plugin-pb-go#4

@disq disq closed this May 30, 2023
@disq disq deleted the fix/specreader-escape-json-embeds branch May 30, 2023 12:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: File Substitution For JSON

3 participants