Add docs for configs of Azure Blob SystemProducer #1323
Add docs for configs of Azure Blob SystemProducer #1323cameronlee314 merged 2 commits intoapache:masterfrom
Conversation
cameronlee314
left a comment
cameronlee314
left a comment
There was a problem hiding this comment.
Thanks for adding the docs.
| |systems.**_system-name_**.<br>bulk.flush.interval.ms|never|How often buffered messages should be flushed.| | ||
|
|
||
| #### <a name="azure-blob-storage"></a>[3.7 Azure Blob Storage](#azure-blob-storage) | ||
| Configs for producing to [Azure Blob Storage](https://azure.microsoft.com/en-us/services/storage/blobs/). This section applies if you have set systems.**__system-name__**.samza.factory = `org.apache.samza.system.azureblob.AzureBlobSystemFactory`.<br> |
There was a problem hiding this comment.
Minor: The **__system-name__** part looks a little inconsistent with the other sections (which use systems.*.samza.factory).
There was a problem hiding this comment.
I did it on purpose to be able to highlight that system-name is the Azure container name. Seemed simpler to do this.
docs/learn/documentation/versioned/jobs/samza-configurations.md
Outdated
Show resolved
Hide resolved
docs/learn/documentation/versioned/jobs/samza-configurations.md
Outdated
Show resolved
Hide resolved
docs/learn/documentation/versioned/jobs/samza-configurations.md
Outdated
Show resolved
Hide resolved
docs/learn/documentation/versioned/jobs/samza-configurations.md
Outdated
Show resolved
Hide resolved
docs/learn/documentation/versioned/jobs/samza-configurations.md
Outdated
Show resolved
Hide resolved
| |systems.**_system-name_**.azureblob.proxy.hostname| |if proxy.use is true then host name of proxy.| | ||
| |systems.**_system-name_**.azureblob.proxy.port| |if proxy.use is true then port of proxy.| | ||
| |samza.azureblob.log.slowRequestMs|30 secs|The duration after which an Azure request will be logged as a warning.| | ||
| |systems.**_system-name_**.azureblob.writer.factory.class|`org.apache.samza.system.`<br>`azureblob.avro.`<br>`AzureBlobAvroWriterFactory`|Fully qualified class name of the `org.apache.samza.system.azureblob.producer.AzureBlobWriter` impl for the system producer.<br><br>The default writer creates blobs that are of type AVRO and require the messages sent to a blob to be AVRO records. The blobs created by the default writer are of type [Block Blobs](https://docs.microsoft.com/en-us/rest/api/storageservices/understanding-block-blobs--append-blobs--and-page-blobs#about-block-blobs).<br>All the following configs are relevant to this default writer.| |
There was a problem hiding this comment.
Regarding "All the following configs are relevant to this default writer.": The following configs apply to other writers too, right? The wording kind of makes it sound like the following configs won't apply to a non-default writer. Can you please clarify that a little bit (or maybe you can just remove that sentence)?
There was a problem hiding this comment.
yes, you are right. if a new non-default writer were to wired it, some of these would apply to it but not all - for example, the new writer might choose to create append blobs and not block blobs in which case flush threshold size and flush timeout and maybe even thread pool count dont make sense as these are all for uploading blocks of a blob. But understand your concern and removing that sentence.
|
FYI, in case you didn't know, you can test what your changes look like by following |
|
Actually, IntelliJ has an optional side bar which shows what my changes look like. |
RB=2024394 G=samza-reviewers R=samza-reviewers,calee A=calee
2 participants
No description provided.