Setting Throughput At The Collection Level

[brushwood24]

I've seen examples of how to set throughput at the container level but I'd like to see how to set it at the collection level. I have tried setting:

"options": { "throughput": 3000 }

in Microsoft.DocumentDb/databaseAccounts/apis/databases/containers

It works when I run the template for the first time but then fails on subsequent runs with the error:

"Throughput update is not allowed through options. Ensure to use throughput settings for updating throughput."

Since ARM templates are supposed to be idempotent, I think I must be missing something here. How should I be doing this?

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: cecb9c8e-133d-065b-314f-ff173f74a3d6
• Version Independent ID: 0d9926b1-569f-d4af-ba21-609393e6d7fc
• Content: Azure Resource Manager templates for Azure Cosmos DB
• Content Source: articles/cosmos-db/manage-sql-with-resource-manager.md
• Service: cosmos-db
• GitHub Login: @markjbrown
• Microsoft Alias: mjbrown

[Mike-Ubezzi-MSFT]

@brushwood24 Thank you for the detailed inquiry, we are investigating and will update you when we have additional information to provided. Is it possible to provide a copy of your ARM template minus any sensitive data such as subscription id. Just enough of it to show how you are passing the options. This will help us isolate the issue. Thank you.

[brushwood24]

Here's an ARM template that demonstrates what I am describing:

example.json.txt

You can run it once and it will create a cosmos db account, database and two collections with the throughput specified in the template. Any subsequent runs will fail with the error:

"Throughput update is not allowed through options. Ensure to use throughput settings for updating throughput."

If you remove line 83:

"options": { "throughput": 400 },

it will run without errors. Is there a different method I should be using to set the throughput (settings vs options)? Regardless, I would expect any ARM template to be idempotent.

[markjbrown]

Investigating. Thanks.

[markjbrown]

Attached is an example of how to update the throughput for a database. We missed creating examples for this.

azuredeploy-sql-update-ru-database.json.txt

Thank you for reporting this. Will have this fully documented very soon with examples for changing throughput for database and containers.

Thanks.

[brushwood24]

I'm actually looking for how to set the throughout at the collection level, not database. Do I do essentially the same thing with a nested settings resource but on the Microsoft.DocumentDb/databaseAccounts/apis/databases/containers resource?

[Mike-Ubezzi-MSFT]

@brushwood24 I believe so, as I was attempting to create an ARM example but the functionality does not allow you to export the template with additional config items made post-deployment but, the following example should get you close:

   		{

		"type": "Microsoft.DocumentDb/databaseAccounts/apis/databases/containers",
		"name": "[concat(variables('accountName'), '/sql/', parameters('databaseName'), '/', parameters('container1Name'))]",
		"apiVersion": "2016-03-31",
		"dependsOn": [ "[resourceId('Microsoft.DocumentDB/databaseAccounts/apis/databases', variables('accountName'), 'sql', parameters('databaseName'))]" ],
		"properties":
		{
			"resource":{
				"id":  "[parameters('container1Name')]",
				"partitionKey": {
					"paths": [
					"/MyPartitionKey1"
					],
					"kind": "Hash"
				},
				"indexingPolicy": {
					"indexingMode": "consistent",
					"includedPaths": [{
							"path": "/*",
							"indexes": [
								{
									"kind": "Range",
									"dataType": "number",
									"precision": -1
								},
								{
									"kind": "Range",
									"dataType": "string",
									"precision": -1
								}
							]
						}
					],
					"excludedPaths": [{
							"path": "/MyPathToNotIndex/*"
						}
					],
                    		"throughput": {
		                    "type": "int",
		                    "defaultValue": 400,
		                    "minValue": 400,
		                    "maxValue": 1000000,
		                    "metadata": {
			                "description": "The throughput for the container"
		                }			
	                },
				}
			}
		}
	},

[markjbrown]

@Mike-Ubezzi-MSFT This is incorrect. See attached for example on setting throughput at the container level.

azuredeploy-sql-container-ru.json.txt

thanks.

[Mike-Ubezzi-MSFT]

@brushwood24 I hope the example provided is a workable solution. Can you verify that running additional updates via ARM is not throwing an error, if you are looking to make changes to throughput via ARM.

[brushwood24]

@Mike-Ubezzi-MSFT / @markjbrown , This looks like the same method I used in my example template that generates the error:

"Throughput update is not allowed through options. Ensure to use throughput settings for updating throughput."

It works the first time the template is run but then errors out for every subsequent run. Am I missing something?

[Mike-Ubezzi-MSFT]

@brushwood24 Thank you for this additional detail. I am reaching out to the PG to get an answer. If you have any additional details (full error exception with GUIDs) that would be helpful if there is nothing obvious that might stand out.

[Mike-Ubezzi-MSFT]

@brushwood24 Please find a link to what should be the correct syntax for making an update to collection throughput. The difference is the use of a parameters.json file for options.

azuredeploy-sql-ru-update.zip

Please let us know if you have additional questions.

[Mike-Ubezzi-MSFT]

@brushwood24 We will now proceed to close this thread. If there are further questions regarding this matter, please comment and we will gladly continue the discussion.

[markjbrown]

This is now fixed. Thanks for reporting. can close this issue now.

thanks.

[aaron-0]

Using a separate template to update the throughput is ok. But usually, you would have a single template that you first use to create a resource and then run again to update.

The current behaviour of the templates makes this difficult. If the template contains the options-->throughput section on the database and the value differs from what is currently deployed you get

"Throughput update is not allowed through options. Ensure to use throughput settings for updating throughput."

If you remove that section and instead just try "Microsoft.DocumentDB/databaseAccounts/apis/databases/settings" for setting throughput in it tells you the property doesn't exist since the db created doesn't have throughput provisioned at the db level.

I've had to resort to a createDatabase Yes/No parameter as a workaround and skip the db creation after the first deploy.

Is there some better way to get around this?

[aaron-0]

I've uploaded an example of how I had to work around this behaviour https://gist.github.com/aaron-0/c69c0af9b29497297e46b5ea51b48abd

I also noted that even though I specify index kinds when you view the settings in the portal you just end up with:
"includedPaths": [ { "path": "/*", "indexes": [] }

[markjbrown]

@aaron-0 Hi and sorry for the delay in responding. If you are trying to set throughput for a database or container use one of these two templates below.

https://azure.microsoft.com/en-us/resources/templates/101-cosmosdb-sql-database-ru-update/
https://azure.microsoft.com/en-us/resources/templates/101-cosmosdb-sql-container-ru-update/

Let me know if you have any other questions. Thanks

[aaron-0]

@markjbrown thanks for the response. The mechanism you have linked for updating is already incorporated into the gist I added: https://gist.github.com/aaron-0/c69c0af9b29497297e46b5ea51b48abd

What I'm trying to achieve is a single idempotent arm template for creating the database initially and then update it over time. e.g when I add a new collection, change an index or change throughput.

Most other azure resources I've created via ARM template support this but the fact cosmos db has one place to set throughput on create(options) and another for update(settings) makes this difficult

[markjbrown]

Yes, I understand your frustration. The reason it is this way is our resource provider does not support PATCH. This is something we are evaluating for the future. I appreciate your feedback on this. It will be incorporated into our discussions on this topic.

btw, another customer has a similar requirement and he came up with a novel solution. Take a look at this blog post of his as it may suit your needs.
https://blog.annejanbarelds.nl/2019/05/21/leveraging-arm-templates-for-deploying-azure-cosmosdb/

Thanks again.

[aaron-0]

Thanks for your help, I'll make use of that solution, as I like it more than what I was doing

[tmbalun17]

I had the same exact problem I am hoping it does not have to be so as I have to keep a readme of what needs to be done for new deployments and then check-in code with isUpdate : true in the source code based on the blog link above for CI CD to keep running.

[joshschmitter]

I agree there should not be a separate template required for updating throughput. This defeats the purpose of idempotent IaC. I would consider this a bug. It's inconsistent with how all other azure resources work and breaks the model of having a parameters and template file pair.

[markjbrown]

Hi @joshschmitter we are actively working to update this now. I should have new ARM templates published in the next 30 days or sooner.

Thanks.

[ericdboyd]

@markjbrown did your last comment mean you are updating the templates in the docs to reflect this deployment trickery, or that the resource provider is being updated so that it is idempotent and you don't have to do this deployment trickery?

[markjbrown]

It's the latter. We're fixing the broken behavior and implementing it such that redeploying the same template with new throughput will update the throughput for the resource.

Thanks.

[Zenuka]

Should this issue be re-opened or is there another issue where I can check the progress?

[markjbrown]

Working to get this out over the next couple of weeks.

[rnowosielski]

@markjbrown is there some place where I could track the work you are describing?

You write:

It's the latter. We're fixing the broken behavior and implementing it such that redeploying the same template with new throughput will update the throughput for the resource.

I would love to have some place where I can come and check the status (whether it is in progress or done, etc) to be able to track the dependencies on my side.

[dannyhorodniczy]

@markjbrown has this been completed? I am no longer receiving this error.

[markjbrown]

Yes, if you are using our latest ARM version then this has been fixed.
https://docs.microsoft.com/en-us/azure/templates/microsoft.documentdb/2019-08-01/databaseaccounts

Thanks everyone for your patience. We are working on updating our docs and sample templates over the next week or two.