Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Support for "Overwrite" instruction in appspec.yml "Files" section #14

Open
colinbjohnson opened this issue Jun 13, 2015 · 92 comments

Comments

Projects
None yet
@colinbjohnson
Copy link

commented Jun 13, 2015

Feature Request:

Would like the ability to have an "overwrite" instruction in the appspec.yml "Files" section.

Use:

In my initial testing, I wanted to clobber all files in /var/www/html with new files. This resulted in a failure because the files already existed. Specifically, the following failure: 2015-06-13 20:58:40 ERROR [codedeploy-agent(7589)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Error during perform: RuntimeError - File already exists at location /var/www/html/index.html - /opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/installer.rb:113:ingenerate_normal_copy'`

Desired Behavior:

I'd like to be able to create an appspec.yml file with the following section:
files:
- source: web
destination: /var/www/html/
overwrite: true
Notice I've added the "overwrite" instruction, which is currently not offered.

Current Workaround:

I've been working around by utilizing the hook section to do file copies. The hooks/shell script workaround is sufficient, rendering this a lower priority feature request.

@suryanarayanan

This comment has been minimized.

Copy link
Contributor

commented Jul 31, 2015

Hi,
Thanks for the feedback. I've added this feature request to our backlog.

@CNG

This comment has been minimized.

Copy link

commented Aug 18, 2015

I would also love this feature.

@madisonmay

This comment has been minimized.

Copy link

commented Aug 18, 2015

Add one more name to the list.

@kingdango

This comment has been minimized.

Copy link

commented Sep 5, 2015

Of course we need this feature!
But it should not be simply overwrite: true -- it should allow for a few options:
overwrite: none (default)
overwrite: purge
overwrite: merge

Purge removes all files in the destination before performing copying the new files.

Merge leaves the destination as-is, adds the files from source and will replace an files in the destination that also exist in the source.

As for now I guess I need to stage my source files in a temp location and then run a script to achieve this. 🐼 😢

@bslatner

This comment has been minimized.

Copy link

commented Dec 16, 2015

+1 for this feature as well.

Side note: the Windows agent does NOT exhibit this behavior. I've been writing several appspecs for Windows services and I've never run into this. Only when we went to deploy to an Ubuntu server for the first time did we see this behavior.

@JoshuaEstes

This comment has been minimized.

Copy link

commented Jan 7, 2016

+1

5 similar comments
@etcsayeghr

This comment has been minimized.

Copy link

commented Jan 27, 2016

+1

@ixaxaar

This comment has been minimized.

Copy link

commented Feb 9, 2016

+1

@vitalyzeme

This comment has been minimized.

Copy link

commented Mar 11, 2016

+1

@praddc

This comment has been minimized.

Copy link

commented Mar 28, 2016

+1

@gabriellhrn

This comment has been minimized.

Copy link

commented Mar 31, 2016

+1

@JoshuaEstes

This comment has been minimized.

Copy link

commented Apr 1, 2016

So, I hope this helps other people find a solution. In my situation I work a lot with Magento so I deal with PHP, nginx. Composer is also tossed in the mix. I found a solution that works for me and for my case. If anyone has any comments or suggestions, please let me know.


I wanted to be have the production directory synced with the repository that I was using so any file modifications would be reflected. Not just updates to files, but delete the files that are no longer required. I think this is what everyone really wants. The solution I found was to deploy the code into a temporary directory (/var/www/release) and then rsync that directory over to the production directory (/var/www/magento).

appspec.yaml

version: 0.0
os: linux
files:
    - source: ./
      destination: /var/www/release/
permissions:
    - object: /var/www/magento/
      owner: www-data
      group: www-data
hooks:
    BeforeInstall:
        - location: scripts/beforeInstall.bash
    AfterInstall:
        - location: scripts/afterInstall.bash

scripts/beforeInstall.bash

#!/usr/bin/env bash

# I want to make sure that the directory is clean and has nothing left over from
# previous deployments. The servers auto scale so the directory may or may not
# exist.
if [ -d /var/www/release ]; then
    rm -rf /var/www/release
fi
mkdir -vp /var/www/release

scripts/afterInstall.bash

#!/usr/bin/env bash

# I have left a few things in here and will explain this further (see below)
rsync --delete-before --verbose --archive --exclude "htdocs/media/" --exclude ".*" --exclude "htdocs/var/" --exclude "htdocs/app/etc/local.xml" /var/www/release/ /var/www/magento/ > /var/log/deploy.log

The rsync command is what allows you to overwrite all the files you need and it also deletes files if they are deleted from the repository. In my case I am deploying a git repo and there are times when extensions/files are removed and those need to be removed on the server.

There are also directories that are excluded from the rsync command. You do not need to sync cache directories and in my case the htdocs/media directory is a mounted S3 bucket and there are lots of files we do not need to delete.

All of the output is put into a deployment log (/var/log/deploy.log) file that I can use in case something goes wrong and can see the exact files deleted and synced over. When I was first experimenting with this, I ran rsync in dry-run mode and checked the log file to make sure files were synced correctly. I HIGHLY recommend you do the same.


Some things I want to try when I make the time, is to rsync files directly from the code-deploy directory. For each deployment, AWS CodeDeploy puts the code into a specific directory which you can reach by some of the environment variables it sets. This should speed up the deploy a little since it isn't putting the files in two places.

There are some things that have been left out of the scripts. Certain directories are chowned with www-data user/group since the files are served used nginx. You may need something similar. The code that places Magento into maintenance mode and removes it from maintenance mode has also been removed since this needs to be generic. You may need similar logic.


You can read more about the various hooks and when they occur in the docs at http://docs.aws.amazon.com/codedeploy/latest/userguide/app-spec-ref.html and also see the environment variables that can be used within the scripts.

@maorhayoun

This comment has been minimized.

Copy link

commented Jul 17, 2016

+1

2 similar comments
@sverdier

This comment has been minimized.

Copy link

commented Aug 17, 2016

+1

@aronhaltamz

This comment has been minimized.

Copy link

commented Aug 17, 2016

+1

@TKLaracuente

This comment has been minimized.

Copy link

commented Aug 31, 2016

@JoshuaEstes You beautiful SOB, this is exactly what I needed.

@jody-greene

This comment has been minimized.

Copy link

commented Aug 31, 2016

+1

@JoshuaEstes

This comment has been minimized.

Copy link

commented Aug 31, 2016

Thanks, glad I could help!

@cesc1989

This comment has been minimized.

Copy link

commented Sep 14, 2016

+1

3 similar comments
@lperezdude

This comment has been minimized.

Copy link

commented Sep 15, 2016

+1

@oswmilanez

This comment has been minimized.

Copy link

commented Oct 5, 2016

+1

@oswmilanez

This comment has been minimized.

Copy link

commented Oct 5, 2016

+1

@knick-burns

This comment has been minimized.

Copy link

commented Oct 11, 2016

This is a good workaround. The issue I have is with Amazon and the fact that it's been over a year and yet this feature is still not implemented.

@Saeven

This comment has been minimized.

Copy link

commented Oct 28, 2016

+1. It's a must.

@agutierrezl

This comment has been minimized.

Copy link

commented Nov 2, 2016

+100

@ashancock

This comment has been minimized.

Copy link

commented Nov 2, 2016

+1

@ratancs

This comment has been minimized.

Copy link

commented Dec 15, 2016

any update on this feature?

@GoodMirek

This comment has been minimized.

Copy link

commented Jan 21, 2017

+1

@jonpchin

This comment has been minimized.

Copy link

commented Jan 23, 2017

Any update on this?

@renatosc

This comment has been minimized.

Copy link

commented Dec 16, 2017

... or they could just add support to overwrite files inside the appspec.yml as this issue is asking for and everything would work fine :)

I don't think it is quite as you say as " all following ones will use those settings.". They will not use whatever settings you used in the first deployment. The automatic deployment is set to (by default?) to not overwrite any existing file that it was not there already.

@thatianand

This comment has been minimized.

Copy link

commented Jan 3, 2018

I tried with overwrite:yes, but didn't work. Any solution for this?

@prasadtoke

This comment has been minimized.

Copy link

commented Jan 24, 2018

@thatianand - You will have to add overwrite flag while using aws createDeployment api.
""overwrite:yes"" will not work.

Pasting this example from above -
aws deploy create-deployment --application-name client_applications
--deployment-group-name client_applications_deployment_group
--github-location commitId=09b0180cf1aa0c952ef00f407b5,repository=GIthub_User/repo_name
--deployment-config-name CodeDeployDefault.OneAtATime
--description "Test Description"
--file-exists-behavior OVERWRITE

@legshort

This comment has been minimized.

Copy link

commented Mar 21, 2018

I wish this option can be configured from appspec.yml cuz sometimes CI/CD does not allow customize deployment configuration, so appspec.yml is the only option to change things.

@rohkat-aws rohkat-aws reopened this Jun 19, 2018

@pandeyprateek

This comment has been minimized.

Copy link

commented Jun 26, 2018

is there a way to pass overwrite using appspec.yml file ? Everytime we have to create deployment manually to overwrite the files and then only codepipeline works

@mtoigo

This comment has been minimized.

Copy link

commented Jun 29, 2018

Yeah... this should really just be an easy to pass option rather than removing files in BeforeInstall:
+1

@kevingimbel

This comment has been minimized.

Copy link

commented Jul 12, 2018

Can this be implemented, please? I am trying to integrate an existing instance into a CodeDeploy workflow and it fails because - since it's a pre-existing server - the files are already there; They should just be overwritten. I really don't like that I have to create a extra script just for this one instance. Especially for "in palce" deployments it would make sense to have an overwrite option, either in the appspec.yml or in the AWS Console.

@AntonChernysh

This comment has been minimized.

Copy link

commented Jul 12, 2018

What you want to do is delete existing files and do a first deployment with codedeploy including those files, once done Al subsequent deployments with codedeploy will overwrite those files with no issues.

@kevingimbel

This comment has been minimized.

Copy link

commented Jul 12, 2018

@AntonChernysh yes, I could do that. But why can't I chose the overwrite option and have CodeDeploy overwrite the files if they exist? This project is not a simple "clear all" project; There are some folders on this instance which need to stay in-place and which are not in the appspec.yml, so I now have to manually delete them one by one.

When I use the AWS Console and create a new Deployment from "CodeBuild -> Deployments -> Create Deployment" I have these options for overwriting:

  • Fail the deployment
    An error is reported and the deployment status is changed to Failed.

  • Overwrite the content
    The file in the application revision is copied to the target location on the instance, replacing the previous file.

  • Retain the content
    The file in the application revision is not copied to the instance. The existing file is kept at the target location and treated as part of the new deployment.

Which means CodeBuild is already capable of overwriting files - it just doesn't have an option via appspec.yml or in the Deployment Group configuration.

@AntonChernysh

This comment has been minimized.

Copy link

commented Jul 12, 2018

@kevingimbel Obviously they don't want to implement this since this issue is open for a long time now.
What I normally do is select "Overwrite the content", delete all files that are part of deployment, and let CodeDeploy take care of updating content on a target env.

You do it only once...

@kevingimbel

This comment has been minimized.

Copy link

commented Jul 12, 2018

@AntonChernysh that's what I am doing now, but this is only possible when manually creating a deployment via CLI or AWS Console - and that's the problem.

IMO it should just be an option in the AppSpec or Deployment Group config.

Anyway, I think you're right regarding this not being implemented given that the issue has been open for 3 years.

@fjhixiuxiu

This comment has been minimized.

Copy link

commented Jul 20, 2018

+1

@annamataws

This comment has been minimized.

Copy link

commented Aug 14, 2018

We will look into a way to allow this setting when the deploy command is not run directly by the user (ex: autoscaling scale up, codepipeline)

@nit-practo

This comment has been minimized.

Copy link

commented Oct 11, 2018

Clearing all files will slow down the launch process if we have asset files which doesn't change frequently. We will be unnecessarily building assets every time which slows down the launch process.

@Lordling

This comment has been minimized.

Copy link

commented Dec 20, 2018

Why is this not an option yet? Without appspec.yml option to specify overwrite behaviour, my auto-scaling groups cannot scale up because when Scale-Up event happens, the new instances do not CodeDeploy with overwrite behaviour!! This causes those launched instances to fail deployment ...

If the option was right there in the appspec.yml file - you know, since that file is supposed to dictate installation behaviour - we would be able to fully integrate code deploy with auto scaling but right now it fails ...

@dlo

This comment has been minimized.

Copy link

commented Feb 11, 2019

Is this package still maintained? Countless open issues like this with presupplied patches are sitting on deaf ears. Would be nice if someone on the AWS side would at least acknowledge the status.

@annamataws

This comment has been minimized.

Copy link

commented Feb 11, 2019

Hi,

This is still a work in progress.

Thank you

@dlo

This comment has been minimized.

Copy link

commented Feb 11, 2019

Thanks, appreciate it @annamataws!

@kcharcast

This comment has been minimized.

Copy link

commented Mar 13, 2019

4 years and still no conclusion?

@miradnan

This comment has been minimized.

Copy link

commented Mar 18, 2019

+1

@waldoj

This comment has been minimized.

Copy link

commented Mar 19, 2019

This is a baffling omission. Please add support for this.

@pkaramol

This comment has been minimized.

Copy link

commented Apr 4, 2019

👍

@AlexL-CC

This comment has been minimized.

Copy link

commented Apr 11, 2019

4 years for such simple request. Shame. +1.

@hoang-innomizetech

This comment has been minimized.

Copy link

commented Apr 20, 2019

I faced this issue and I have to update script to clean directory to avoid this error

@icf-chartmann

This comment has been minimized.

Copy link

commented Apr 26, 2019

Well... even though this is 4 years in the making, at least I know other people are experience the same heartache that I am.

@krish-sie

This comment has been minimized.

Copy link

commented May 14, 2019

+1
From last four years no update for this small request. we need to some file configuration options other than using scripts in hooks. Can someone please prioritize this.?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.