Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(codecs) csv documentation defaults as ascii_char #20498

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open

fix(codecs) csv documentation defaults as ascii_char #20498

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
99c08d7
set type_override for ascii_char
scMarkus May 15, 2024
467189d
specially handle "docs::type_override" with "ascii_char" in generate-…
scMarkus Jun 25, 2024
1add8a0
Apply suggestions from code review
scMarkus Jun 25, 2024
a646800
add missing filds in csv.rs and coresponsing cue docs as well as shor…
scMarkus Jun 27, 2024
24fc1a7
fix spelling
scMarkus Jun 27, 2024
fe647da
Update scripts/generate-component-docs.rb
scMarkus Jun 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions changelog.d/18669_csv_documentation_u8_display.fix.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
Fixing the representation of the ascii character configuration in the csv encoder documentation so
that they are shown as human readable characters contrary to integers.

author scMarkus
3 changes: 3 additions & 0 deletions lib/codecs/src/encoding/format/csv.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,7 @@ impl CsvSerializerConfig {
#[derive(Debug, Clone)]
pub struct CsvSerializerOptions {
/// The field delimiter to use when writing CSV.
#[configurable(metadata(docs::type_override = "ascii_char"))]
#[serde(
default = "default_delimiter",
with = "vector_core::serde::ascii_char",
Expand All @@ -98,6 +99,7 @@ pub struct CsvSerializerOptions {
/// like \ (instead of escaping quotes by doubling them).
///
/// To use this, `double_quotes` needs to be disabled as well otherwise it is ignored.
#[configurable(metadata(docs::type_override = "ascii_char"))]
#[serde(
default = "default_escape",
with = "vector_core::serde::ascii_char",
Expand All @@ -106,6 +108,7 @@ pub struct CsvSerializerOptions {
pub escape: u8,

/// The quote character to use when writing CSV.
#[configurable(metadata(docs::type_override = "ascii_char"))]
#[serde(
default = "default_escape",
with = "vector_core::serde::ascii_char",
Expand Down
12 changes: 11 additions & 1 deletion scripts/generate-component-docs.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -744,9 +744,19 @@ def resolve_schema(root_schema, schema)
#
# We intentially set no actual definition for these types, relying on the documentation generation
# process to provide the actual details. We only need to specify the custom type name.
#
# For `u8` fields that have a type override of `ascii_char` type, we specifically use the
# character representation of the default value, which ensures that we display the default
# values as an actual string character instead of an ordinal value.
#
# See `lib/codecs/src/encoding/format/csv.rs` for an example.
type_override = get_schema_metadata(schema, 'docs::type_override')
if !type_override.nil?
resolved = { 'type' => { type_override.to_s => {} } }
if type_override == 'ascii_char'
resolved = { 'type' => { type_override.to_s => { 'default' => schema['default'].chr } } }
else
resolved = { 'type' => { type_override.to_s => {} } }
end
description = get_rendered_description_from_schema(schema)
resolved['description'] = description unless description.empty?
return resolved
Expand Down
8 changes: 4 additions & 4 deletions website/content/en/highlights/2024-06-17-0-39-0-upgrade-guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,10 @@ We cover them below to help you upgrade quickly:

#### Path coalescing has been removed {#path-coalescing}

Coalescing of field lookup paths (e.g. `.(field1|field2)`) was deprecated and is now removed This
Coalescing of field lookup paths (e.g., `.(field1|field2)`) was deprecated and is now removed. This
feature did not seem to be used much and significantly complicates parts of the codebase.

If you were using this feature, you can accomplish the same through conditionals in VRL like:
If you were using this feature, you can accomplish the same through conditionals in VRL:

```vrl
field = if exists(.field1) {
Expand All @@ -37,6 +37,6 @@ field = if exists(.field1) {

#### Removal of `enterprise` configuration {#enterprise-configuration}

The `enterprise` global configuration option that was used to integrate Vector with the Datadog has been removed.
Observability Pipelines product was deprecated and has been removed. Please reach out to Datadog
The `enterprise` global configuration option that was used to integrate Vector with Datadog has been removed.
The Observability Pipelines product was deprecated and has been removed. Please reach out to Datadog
support if you are impacted by this.
6 changes: 3 additions & 3 deletions website/cue/reference/components/sinks/base/amqp.cue
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -157,7 +157,7 @@ base: components: sinks: amqp: configuration: {
delimiter: {
description: "The field delimiter to use when writing CSV."
required: false
type: uint: default: 44
type: ascii_char: default: ","
}
double_quote: {
description: """
Expand All @@ -179,7 +179,7 @@ base: components: sinks: amqp: configuration: {
To use this, `double_quotes` needs to be disabled as well otherwise it is ignored.
"""
required: false
type: uint: default: 34
type: ascii_char: default: "\""
}
fields: {
description: """
Expand All @@ -197,7 +197,7 @@ base: components: sinks: amqp: configuration: {
quote: {
description: "The quote character to use when writing CSV."
required: false
type: uint: default: 34
type: ascii_char: default: "\""
}
quote_style: {
description: "The quoting style to use when writing CSV data."
Expand Down
6 changes: 3 additions & 3 deletions website/cue/reference/components/sinks/base/aws_cloudwatch_logs.cue
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -333,7 +333,7 @@ base: components: sinks: aws_cloudwatch_logs: configuration: {
delimiter: {
description: "The field delimiter to use when writing CSV."
required: false
type: uint: default: 44
type: ascii_char: default: ","
}
double_quote: {
description: """
Expand All @@ -355,7 +355,7 @@ base: components: sinks: aws_cloudwatch_logs: configuration: {
To use this, `double_quotes` needs to be disabled as well otherwise it is ignored.
"""
required: false
type: uint: default: 34
type: ascii_char: default: "\""
}
fields: {
description: """
Expand All @@ -373,7 +373,7 @@ base: components: sinks: aws_cloudwatch_logs: configuration: {
quote: {
description: "The quote character to use when writing CSV."
required: false
type: uint: default: 34
type: ascii_char: default: "\""
}
quote_style: {
description: "The quoting style to use when writing CSV data."
Expand Down
6 changes: 3 additions & 3 deletions website/cue/reference/components/sinks/base/aws_kinesis_firehose.cue
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -312,7 +312,7 @@ base: components: sinks: aws_kinesis_firehose: configuration: {
delimiter: {
description: "The field delimiter to use when writing CSV."
required: false
type: uint: default: 44
type: ascii_char: default: ","
}
double_quote: {
description: """
Expand All @@ -334,7 +334,7 @@ base: components: sinks: aws_kinesis_firehose: configuration: {
To use this, `double_quotes` needs to be disabled as well otherwise it is ignored.
"""
required: false
type: uint: default: 34
type: ascii_char: default: "\""
}
fields: {
description: """
Expand All @@ -352,7 +352,7 @@ base: components: sinks: aws_kinesis_firehose: configuration: {
quote: {
description: "The quote character to use when writing CSV."
required: false
type: uint: default: 34
type: ascii_char: default: "\""
}
quote_style: {
description: "The quoting style to use when writing CSV data."
Expand Down
6 changes: 3 additions & 3 deletions website/cue/reference/components/sinks/base/aws_kinesis_streams.cue
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -312,7 +312,7 @@ base: components: sinks: aws_kinesis_streams: configuration: {
delimiter: {
description: "The field delimiter to use when writing CSV."
required: false
type: uint: default: 44
type: ascii_char: default: ","
}
double_quote: {
description: """
Expand All @@ -334,7 +334,7 @@ base: components: sinks: aws_kinesis_streams: configuration: {
To use this, `double_quotes` needs to be disabled as well otherwise it is ignored.
"""
required: false
type: uint: default: 34
type: ascii_char: default: "\""
}
fields: {
description: """
Expand All @@ -352,7 +352,7 @@ base: components: sinks: aws_kinesis_streams: configuration: {
quote: {
description: "The quote character to use when writing CSV."
required: false
type: uint: default: 34
type: ascii_char: default: "\""
}
quote_style: {
description: "The quoting style to use when writing CSV data."
Expand Down
6 changes: 3 additions & 3 deletions website/cue/reference/components/sinks/base/aws_s3.cue
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -421,7 +421,7 @@ base: components: sinks: aws_s3: configuration: {
delimiter: {
description: "The field delimiter to use when writing CSV."
required: false
type: uint: default: 44
type: ascii_char: default: ","
}
double_quote: {
description: """
Expand All @@ -443,7 +443,7 @@ base: components: sinks: aws_s3: configuration: {
To use this, `double_quotes` needs to be disabled as well otherwise it is ignored.
"""
required: false
type: uint: default: 34
type: ascii_char: default: "\""
}
fields: {
description: """
Expand All @@ -461,7 +461,7 @@ base: components: sinks: aws_s3: configuration: {
quote: {
description: "The quote character to use when writing CSV."
required: false
type: uint: default: 34
type: ascii_char: default: "\""
}
quote_style: {
description: "The quoting style to use when writing CSV data."
Expand Down
6 changes: 3 additions & 3 deletions website/cue/reference/components/sinks/base/aws_sns.cue
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -243,7 +243,7 @@ base: components: sinks: aws_sns: configuration: {
delimiter: {
description: "The field delimiter to use when writing CSV."
required: false
type: uint: default: 44
type: ascii_char: default: ","
}
double_quote: {
description: """
Expand All @@ -265,7 +265,7 @@ base: components: sinks: aws_sns: configuration: {
To use this, `double_quotes` needs to be disabled as well otherwise it is ignored.
"""
required: false
type: uint: default: 34
type: ascii_char: default: "\""
}
fields: {
description: """
Expand All @@ -283,7 +283,7 @@ base: components: sinks: aws_sns: configuration: {
quote: {
description: "The quote character to use when writing CSV."
required: false
type: uint: default: 34
type: ascii_char: default: "\""
}
quote_style: {
description: "The quoting style to use when writing CSV data."
Expand Down
6 changes: 3 additions & 3 deletions website/cue/reference/components/sinks/base/aws_sqs.cue
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -243,7 +243,7 @@ base: components: sinks: aws_sqs: configuration: {
delimiter: {
description: "The field delimiter to use when writing CSV."
required: false
type: uint: default: 44
type: ascii_char: default: ","
}
double_quote: {
description: """
Expand All @@ -265,7 +265,7 @@ base: components: sinks: aws_sqs: configuration: {
To use this, `double_quotes` needs to be disabled as well otherwise it is ignored.
"""
required: false
type: uint: default: 34
type: ascii_char: default: "\""
}
fields: {
description: """
Expand All @@ -283,7 +283,7 @@ base: components: sinks: aws_sqs: configuration: {
quote: {
description: "The quote character to use when writing CSV."
required: false
type: uint: default: 34
type: ascii_char: default: "\""
}
quote_style: {
description: "The quoting style to use when writing CSV data."
Expand Down
6 changes: 3 additions & 3 deletions website/cue/reference/components/sinks/base/azure_blob.cue
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -275,7 +275,7 @@ base: components: sinks: azure_blob: configuration: {
delimiter: {
description: "The field delimiter to use when writing CSV."
required: false
type: uint: default: 44
type: ascii_char: default: ","
}
double_quote: {
description: """
Expand All @@ -297,7 +297,7 @@ base: components: sinks: azure_blob: configuration: {
To use this, `double_quotes` needs to be disabled as well otherwise it is ignored.
"""
required: false
type: uint: default: 34
type: ascii_char: default: "\""
}
fields: {
description: """
Expand All @@ -315,7 +315,7 @@ base: components: sinks: azure_blob: configuration: {
quote: {
description: "The quote character to use when writing CSV."
required: false
type: uint: default: 34
type: ascii_char: default: "\""
}
quote_style: {
description: "The quoting style to use when writing CSV data."
Expand Down
6 changes: 3 additions & 3 deletions website/cue/reference/components/sinks/base/console.cue
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,7 @@ base: components: sinks: console: configuration: {
delimiter: {
description: "The field delimiter to use when writing CSV."
required: false
type: uint: default: 44
type: ascii_char: default: ","
}
double_quote: {
description: """
Expand All @@ -163,7 +163,7 @@ base: components: sinks: console: configuration: {
To use this, `double_quotes` needs to be disabled as well otherwise it is ignored.
"""
required: false
type: uint: default: 34
type: ascii_char: default: "\""
}
fields: {
description: """
Expand All @@ -181,7 +181,7 @@ base: components: sinks: console: configuration: {
quote: {
description: "The quote character to use when writing CSV."
required: false
type: uint: default: 34
type: ascii_char: default: "\""
}
quote_style: {
description: "The quoting style to use when writing CSV data."
Expand Down
6 changes: 3 additions & 3 deletions website/cue/reference/components/sinks/base/databend.cue
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,7 @@ base: components: sinks: databend: configuration: {
delimiter: {
description: "The field delimiter to use when writing CSV."
required: false
type: uint: default: 44
type: ascii_char: default: ","
}
double_quote: {
description: """
Expand All @@ -182,7 +182,7 @@ base: components: sinks: databend: configuration: {
To use this, `double_quotes` needs to be disabled as well otherwise it is ignored.
"""
required: false
type: uint: default: 34
type: ascii_char: default: "\""
}
fields: {
description: """
Expand All @@ -200,7 +200,7 @@ base: components: sinks: databend: configuration: {
quote: {
description: "The quote character to use when writing CSV."
required: false
type: uint: default: 34
type: ascii_char: default: "\""
}
quote_style: {
description: "The quoting style to use when writing CSV data."
Expand Down
6 changes: 3 additions & 3 deletions website/cue/reference/components/sinks/base/file.cue
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,7 @@ base: components: sinks: file: configuration: {
delimiter: {
description: "The field delimiter to use when writing CSV."
required: false
type: uint: default: 44
type: ascii_char: default: ","
}
double_quote: {
description: """
Expand All @@ -183,7 +183,7 @@ base: components: sinks: file: configuration: {
To use this, `double_quotes` needs to be disabled as well otherwise it is ignored.
"""
required: false
type: uint: default: 34
type: ascii_char: default: "\""
}
fields: {
description: """
Expand All @@ -201,7 +201,7 @@ base: components: sinks: file: configuration: {
quote: {
description: "The quote character to use when writing CSV."
required: false
type: uint: default: 34
type: ascii_char: default: "\""
}
quote_style: {
description: "The quoting style to use when writing CSV data."
Expand Down
6 changes: 3 additions & 3 deletions website/cue/reference/components/sinks/base/gcp_chronicle_unstructured.cue
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -210,7 +210,7 @@ base: components: sinks: gcp_chronicle_unstructured: configuration: {
delimiter: {
description: "The field delimiter to use when writing CSV."
required: false
type: uint: default: 44
type: ascii_char: default: ","
}
double_quote: {
description: """
Expand All @@ -232,7 +232,7 @@ base: components: sinks: gcp_chronicle_unstructured: configuration: {
To use this, `double_quotes` needs to be disabled as well otherwise it is ignored.
"""
required: false
type: uint: default: 34
type: ascii_char: default: "\""
}
fields: {
description: """
Expand All @@ -250,7 +250,7 @@ base: components: sinks: gcp_chronicle_unstructured: configuration: {
quote: {
description: "The quote character to use when writing CSV."
required: false
type: uint: default: 34
type: ascii_char: default: "\""
}
quote_style: {
description: "The quoting style to use when writing CSV data."
Expand Down
Loading
Loading