Correct template file for AWS_BACKUP_MULTIPART_CHUNK_SIZE env

[drubin]

Fixes #1293

Correctly set up and configure the template in order to use chunked
upload.

It seems that @solidnerd introduced this feature in 607df08 (thanks we require this as well)

But it doesn't appear to work. #1293 and we are also having issues with larger uploads

I have simply enabled parsing this file as the docs recommend https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/gitlab.yml.example#L544

I hope this is correct? Else please point me in the right direction.

[solidnerd]

It would be neat and for backwards compability to remove the multipart_chunk_size when AWS_BACKUP_MULTIPART_CHUNK_SIZE is not set. I think not everyone needs this directly.

[solidnerd]

Hey @drubin,
thanks for you work. Yeah It seems that I missed to implement it in directly. I made a short review to that I hope it helps. If not feel free to ask.

[drubin]

@solidnerd Thanks for the incredibly fast reply. I am not exactly sure how to do "if" statements in the current templating language. Could it be as simple as a sed replace to remove the # comment if the var is set?

I have tried updating the commit to conditionally remove the multipart section if the ENV var is not passed in. I hope this is ok.

It appears that Fog already has default https://github.com/fog/fog-aws/blob/master/lib/fog/aws/models/storage/file.rb#L214 which is lower than the gitlab default? Either way, this seems like a good option to allow us to configure it.

[solidnerd]

Hey @drubin,

thanks for your changes. I will test it later how it's working directly for the next release. I think everything is working as excepted.

The fog-aws default is ~5Mb (5242880B).

I will let you know if some changes are required again or not.

[drubin]

@solidnerd Sure any time just ping me if you require any changes.

Would you prefer I rebase/squash my commits into 1?

[drubin]

I finally got the docker-compose setup installed, it seems as if my regex is not actually removing the inside blocks between the #start-aws-multipart and #end-aws-multipart when the AWS_BACKUP_MULTIPART_CHUNK_SIZE is not set but when I run them inside the container as a simple sed command they work exactly as expected.

Am I doing something wrong?

[solidnerd]

Hey @drubin,
i tried it by my self and get the same problem atm I have no clue why it's not working correctly. We need to dig deeper.

If prefer when the feature is complete a rebase to be easier revertable with the changes and are more clean history.

[drubin]

@solidnerd What happens if we default the value to "null" instead of leaving it blank it would have the same effect no?

[solidnerd]

I think the value of the variable isn't the Problem. What I See now is that in your If condition your variables needs to be qouted so i will look like this `"${AWS_BACKUP_MULTIPART_CHUNK_SIZE}"` Am 29.12.2017 15:14 schrieb "David Rubin" <notifications@github.com>:

…

@solidnerd <https://github.com/solidnerd> What happens if we default the value to "null" instead of leaving it blank it would have the same effect no? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1449 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA2Gb9ONbijvU3ZaUizZui9NfNEAs9n8ks5tFPO4gaJpZM4RM-YF> .

[solidnerd]

Ah okay Email replies doesn't support Markdown "${AWS_BACKUP_MULTIPART_CHUNK_SIZE}"

[drubin]

@solidnerd That isn't the issue because I added debug statements inside the IF statement from the functions.

I am going to keep trying to figure this out as this is something I would really like to see.

What is the policy about submitting releases that don't directly correspond to gitlab releases? Or would this fix have to wait for the next gitlab release?

[solidnerd]

Nope there is no need to wait for the next GitLab release. This is a fix of the current behaviour so a release cloud have an own Tag (10.3.2-1) or it will live on latest until the next GitLab Release ist available. Am 29.12.2017 15:36 schrieb "David Rubin" <notifications@github.com>: @solidnerd <https://github.com/solidnerd> That isn't the issue because I added debug statements inside the IF statement from the functions. I am going to keep trying to figure this out as this is something I would really like to see. What is the policy about submitting releases that don't directly correspond to gitlab releases? Or would this fix have to wait for the next gitlab release? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1449 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA2GbyYxJakBYnTae9epoRywT7ZFyX2Qks5tFPjbgaJpZM4RM-YF> .

[drubin]

@solidnerd, in that case, would you mind me adding a flag for encryption: 'AES256' as well?

[solidnerd]

Feel free to do it. Also the storage_class is missing. For more information have a look in the upstream https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/gitlab.yml.example#L547

[drubin]

@solidnerd I managed to get it working, I think there was an issue with the #start-aws starting the #start-aws-multipart LOTS of debugging later I changed the variable.

I have also added config options for AES encryption as well as storage class options and updated the readme.

Let me know if it works for you, or if you would like any other changes

[solidnerd]

It will be in the next release we didn't a separate release for this because 10.3.3 is available now.

[drubin]

Thanks