local vs. remote State

[SourceSign]

Hi all,

first of all I would like to thank you for using Pulumi. It is an awsome tool. However, I have made an observation that I would like to share with you. Maybe you have some advice for me.

I used the Pulumi Automation API to write a CLI tool that creates an S3 bucket and then uploads files for a web page to that bucket. This is essentially an extension of the example for Pulumi inline programs. The state is also stored in an S3 bucket.

In doing this, I've noticed a performance issue. The more files I try to upload to the bucket, the slower the upload through Pulumi becomes. If I upload a web page with 10 files it takes me 19s, if I upload a web page with 500 files it takes me 10 min and if I upload 2000 files it takes me 1h 40 min.

If I then change the index.html (only one file) and start an upsert again, Pulumi needs 10s for 10 files, 6 min for 500 files and 58 min for 2000 files.

If I save the state locally and also upload the content, I get much better results, with 500 files 33s, with 2000 files 8m 49s.

This behavior is shown in the attached graphic:

Is this behavior known/desired? Is there any recommendation.

[Frassle]

This is probably due to the O(N²) performance of the pulumi state system. It's not that the S3 files are taking longer, its that pulumi is taking longer to record in it's state that it's uploaded all those S3 files.

Saving the state to a local file while still O(N²) has much lower constant factors (local disk is orders of magnitude faster than the network), thus the speed increase you see while using local state.

[SourceSign]

Thanks for the answer. Is this quadratic behavior documented somewhere or is this just the experience of users?

[Frassle]

We have an issue about it #3930