Cloud storage objects upload

[davidebelloni]

Can you provide an example about uploading an object to a Cloud Storage bucket?

Thanks

[krwenholz]

Hi Davide,

You could probably hack something into Deployment Manager using a feature we're working on (documentation coming sometime Q4), but uploading objects to Cloud Storage isn't really a supported use case. Your best tool for that is gcloud or the API.

What's your particular use case though? Is this file the same for every deployment? Are you referencing a GCS object in your deployment template?

[davidebelloni]

Hi,
I've a project that hosts golden images for others projects and (in a GCS bucket) their startup scripts.
I'm managing project and bucket with DM and images with Packer.
I can manage this GCS objects with gcloud or gsutil, but with something like DM the overall process should be more clear.

[aljim]

As a workaround you can define a cloud function that does that for you, here is a sample to call a cloud function within DM

https://github.com/GoogleCloudPlatform/deploymentmanager-samples/tree/master/examples/v2/cloud_functions

[katyp]

Closing this issue. Please re-open if more is needed.

[krwenholz]

Hi Davide,

In the sample Alejandro linked to, you can see DM calling the raw Cloud Function. If the content you wanted was in your deployment, you could get it into Cloud Storage using their insert API in a similar way:

https://cloud.google.com/storage/docs/json_api/v1/objects/insert

You would need to do some configuring to get rid of that object on deletion of the deployment though (something we don't have docs for yet). Alternatively, you could copy objects between buckets with something like the following

resources:
- # All your existing resources.
...
- name: some-name
  action: gcp-types/storage-v1:storage.objects.copy
  metadata:
    runtimePolicy:
    - CREATE
  properties:
    sourceBucket: my-source-bucket
    sourceObject: object-name
    destinationBucket: us-central1-bucket # Put the proper bucket name here.
    destinationObject: object-name

You would want to also add an action for deleting the object. It would look like the above but use a DELETE runtime policy and call the object delete API. Other acceptable policies are UPDATE_ALWAYS and UPDATE_ON_CHANGE (change of the deployment, not the object).

Hopefully that helps and gets you going in the right direction. Let us know if there's anything else we can do to help.

Happy Monday.

[gadaldo]

Hi,

what about deleting a bucket with content in it (such as a previously created staging location)?

Regards

[krwenholz]

Hi @gadaldo,

There isn't an API to delete a bucket with content, so you would need to delete the contents first. This can be done via gcloud or the console. If you want to do the whole thing with Deployment Manager, I would recommend looking into running a a Cloud Container Builder script in DM.

Hopefully that helps.

Happy Monday,

Kyle

[gadaldo]

Oh so there is not a way of say:

- name: {{ PROJECT_NAME }}-event-dataflow
  type: storage.v1.bucket
  properties:
    location: europe-west2
    storageClass: REGIONAL
    deleteContents: True

cause I am trying but no luck.

Regards

[krwenholz]

Hi @gadaldo ,

Yes, this is a limitation of the Google Cloud Storage API. Deployment Manager can only interact with APIs, so there is no declarative way for us to delete bucket contents, hence the suggestion to use a gcloud script like described in our Cloud Container Builder example (linked above).

Best,

Kyle

[gadaldo]

Ok thank you for replying. We are running out of python so I'll go via gcloud command line for this object deletion in my containerized deployment project. The idea was to have a completely idempotent deployment project with no dependencies (only docker) and as much as declarative as possible.

thank you for your reply

Happy monday

Giuseppe