Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Support for "Overwrite" instruction in appspec.yml "Files" section #14

Closed
colinbjohnson opened this issue Jun 13, 2015 · 173 comments
Closed

Comments

@colinbjohnson
Copy link

@colinbjohnson colinbjohnson commented Jun 13, 2015

Feature Request:

Would like the ability to have an "overwrite" instruction in the appspec.yml "Files" section.

Use:

In my initial testing, I wanted to clobber all files in /var/www/html with new files. This resulted in a failure because the files already existed. Specifically, the following failure: 2015-06-13 20:58:40 ERROR [codedeploy-agent(7589)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Error during perform: RuntimeError - File already exists at location /var/www/html/index.html - /opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/installer.rb:113:ingenerate_normal_copy'`

Desired Behavior:

I'd like to be able to create an appspec.yml file with the following section:
files:
- source: web
destination: /var/www/html/
overwrite: true
Notice I've added the "overwrite" instruction, which is currently not offered.

Current Workaround:

I've been working around by utilizing the hook section to do file copies. The hooks/shell script workaround is sufficient, rendering this a lower priority feature request.

@suryanarayanan
Copy link
Contributor

@suryanarayanan suryanarayanan commented Jul 31, 2015

Hi,
Thanks for the feedback. I've added this feature request to our backlog.

@CNG
Copy link

@CNG CNG commented Aug 18, 2015

I would also love this feature.

@madisonmay
Copy link

@madisonmay madisonmay commented Aug 18, 2015

Add one more name to the list.

@kingdango
Copy link

@kingdango kingdango commented Sep 5, 2015

Of course we need this feature!
But it should not be simply overwrite: true -- it should allow for a few options:
overwrite: none (default)
overwrite: purge
overwrite: merge

Purge removes all files in the destination before performing copying the new files.

Merge leaves the destination as-is, adds the files from source and will replace an files in the destination that also exist in the source.

As for now I guess I need to stage my source files in a temp location and then run a script to achieve this. 🐼 😢

@bslatner
Copy link

@bslatner bslatner commented Dec 16, 2015

+1 for this feature as well.

Side note: the Windows agent does NOT exhibit this behavior. I've been writing several appspecs for Windows services and I've never run into this. Only when we went to deploy to an Ubuntu server for the first time did we see this behavior.

@JoshuaEstes
Copy link

@JoshuaEstes JoshuaEstes commented Jan 7, 2016

+1

5 similar comments
@etcsayeghr
Copy link

@etcsayeghr etcsayeghr commented Jan 27, 2016

+1

@ixaxaar
Copy link

@ixaxaar ixaxaar commented Feb 9, 2016

+1

@vitalyzeme
Copy link

@vitalyzeme vitalyzeme commented Mar 11, 2016

+1

@praddc
Copy link

@praddc praddc commented Mar 28, 2016

+1

@gabriellhrn
Copy link

@gabriellhrn gabriellhrn commented Mar 31, 2016

+1

@JoshuaEstes
Copy link

@JoshuaEstes JoshuaEstes commented Apr 1, 2016

So, I hope this helps other people find a solution. In my situation I work a lot with Magento so I deal with PHP, nginx. Composer is also tossed in the mix. I found a solution that works for me and for my case. If anyone has any comments or suggestions, please let me know.


I wanted to be have the production directory synced with the repository that I was using so any file modifications would be reflected. Not just updates to files, but delete the files that are no longer required. I think this is what everyone really wants. The solution I found was to deploy the code into a temporary directory (/var/www/release) and then rsync that directory over to the production directory (/var/www/magento).

appspec.yaml

version: 0.0
os: linux
files:
    - source: ./
      destination: /var/www/release/
permissions:
    - object: /var/www/magento/
      owner: www-data
      group: www-data
hooks:
    BeforeInstall:
        - location: scripts/beforeInstall.bash
    AfterInstall:
        - location: scripts/afterInstall.bash

scripts/beforeInstall.bash

#!/usr/bin/env bash

# I want to make sure that the directory is clean and has nothing left over from
# previous deployments. The servers auto scale so the directory may or may not
# exist.
if [ -d /var/www/release ]; then
    rm -rf /var/www/release
fi
mkdir -vp /var/www/release

scripts/afterInstall.bash

#!/usr/bin/env bash

# I have left a few things in here and will explain this further (see below)
rsync --delete-before --verbose --archive --exclude "htdocs/media/" --exclude ".*" --exclude "htdocs/var/" --exclude "htdocs/app/etc/local.xml" /var/www/release/ /var/www/magento/ > /var/log/deploy.log

The rsync command is what allows you to overwrite all the files you need and it also deletes files if they are deleted from the repository. In my case I am deploying a git repo and there are times when extensions/files are removed and those need to be removed on the server.

There are also directories that are excluded from the rsync command. You do not need to sync cache directories and in my case the htdocs/media directory is a mounted S3 bucket and there are lots of files we do not need to delete.

All of the output is put into a deployment log (/var/log/deploy.log) file that I can use in case something goes wrong and can see the exact files deleted and synced over. When I was first experimenting with this, I ran rsync in dry-run mode and checked the log file to make sure files were synced correctly. I HIGHLY recommend you do the same.


Some things I want to try when I make the time, is to rsync files directly from the code-deploy directory. For each deployment, AWS CodeDeploy puts the code into a specific directory which you can reach by some of the environment variables it sets. This should speed up the deploy a little since it isn't putting the files in two places.

There are some things that have been left out of the scripts. Certain directories are chowned with www-data user/group since the files are served used nginx. You may need something similar. The code that places Magento into maintenance mode and removes it from maintenance mode has also been removed since this needs to be generic. You may need similar logic.


You can read more about the various hooks and when they occur in the docs at http://docs.aws.amazon.com/codedeploy/latest/userguide/app-spec-ref.html and also see the environment variables that can be used within the scripts.

@maorhayoun
Copy link

@maorhayoun maorhayoun commented Jul 17, 2016

+1

2 similar comments
@sverdier
Copy link

@sverdier sverdier commented Aug 17, 2016

+1

@aronhaltamz
Copy link

@aronhaltamz aronhaltamz commented Aug 17, 2016

+1

@TKLaracuente
Copy link

@TKLaracuente TKLaracuente commented Aug 31, 2016

@JoshuaEstes You beautiful SOB, this is exactly what I needed.

@jody-greene
Copy link

@jody-greene jody-greene commented Aug 31, 2016

+1

@JoshuaEstes
Copy link

@JoshuaEstes JoshuaEstes commented Aug 31, 2016

Thanks, glad I could help!

@cesc1989
Copy link

@cesc1989 cesc1989 commented Sep 14, 2016

+1

3 similar comments
@lperezdude
Copy link

@lperezdude lperezdude commented Sep 15, 2016

+1

@oswmilanez
Copy link

@oswmilanez oswmilanez commented Oct 5, 2016

+1

@oswmilanez
Copy link

@oswmilanez oswmilanez commented Oct 5, 2016

+1

@knick-burns
Copy link

@knick-burns knick-burns commented Oct 11, 2016

This is a good workaround. The issue I have is with Amazon and the fact that it's been over a year and yet this feature is still not implemented.

@Saeven
Copy link

@Saeven Saeven commented Oct 28, 2016

+1. It's a must.

@agutierrezl
Copy link

@agutierrezl agutierrezl commented Nov 2, 2016

+100

@ashancock
Copy link

@ashancock ashancock commented Nov 2, 2016

+1

@ratancs
Copy link

@ratancs ratancs commented Dec 15, 2016

any update on this feature?

@GoodMirek
Copy link

@GoodMirek GoodMirek commented Jan 21, 2017

+1

@jonpchin
Copy link

@jonpchin jonpchin commented Jan 23, 2017

Any update on this?

@waldoj
Copy link

@waldoj waldoj commented Feb 10, 2021

Thank you, @philstrong!

@engineerakki
Copy link

@engineerakki engineerakki commented Mar 9, 2021

@philstrong
Was this picked up in your sprint ?
Is there any official timeline for this feature ?

@quentinhayot
Copy link

@quentinhayot quentinhayot commented Mar 10, 2021

Is this really happening ?!!!
2021 WILL be a good year after all ! :')

@dreaddesign
Copy link

@dreaddesign dreaddesign commented Mar 10, 2021

if this is fixed, i would be very happy. :) +1

@philstrong
Copy link
Contributor

@philstrong philstrong commented Mar 11, 2021

@philstrong
Copy link
Contributor

@philstrong philstrong commented Mar 26, 2021

This will be coming soon in the next release of the agent. @chrisdibble did all of the work so he can explain the feature/fix a little better.

@waldoj
Copy link

@waldoj waldoj commented Mar 26, 2021

Great news! Thanks so much for your attention to this.

@chrisdibble
Copy link

@chrisdibble chrisdibble commented Mar 26, 2021

Not a huge change - we're just allowing the file_exists_behavior to be set as part of the appspec file, which will take precedence over the corresponding command line option for file_exists_behavior for the upcoming release. We're not allowing fine grained control of file_exists_behavior per file/directory listed in the appspec files section, though. We'll consider that if demand from customers still exists following the upcoming release.

@0cool-f
Copy link

@0cool-f 0cool-f commented Mar 29, 2021

Not a huge change - we're just allowing the file_exists_behavior to be set as part of the appspec file, which will take precedence over the corresponding command line option for file_exists_behavior for the upcoming release. We're not allowing fine grained control of file_exists_behavior per file/directory listed in the appspec files section, though. We'll consider that if demand from customers still exists following the upcoming release.

Thanks a lot for fixing it. You may not have noticed that, but this is exactly THE huge fix thousands of people have been waiting for.

@alinoudev
Copy link

@alinoudev alinoudev commented Apr 3, 2021

On what date its planned to release?

@quentinhayot
Copy link

@quentinhayot quentinhayot commented Apr 6, 2021

Not a huge change - we're just allowing the file_exists_behavior to be set as part of the appspec file, which will take precedence over the corresponding command line option for file_exists_behavior for the upcoming release. We're not allowing fine grained control of file_exists_behavior per file/directory listed in the appspec files section, though. We'll consider that if demand from customers still exists following the upcoming release.

Guys, this is it !

On what date its planned to release?

Hopefully soon ! I'll just put the file_exists_behavior in the appspec already...

@ghost
Copy link

@ghost ghost commented Apr 10, 2021

Looking forward to this!

@s-larbi
Copy link

@s-larbi s-larbi commented Apr 30, 2021

Any updates regarding the overwrite feature? It takes time to delete everything and setup apps from scratch. The feature would be a great time saver.

@philstrong
Copy link
Contributor

@philstrong philstrong commented May 2, 2021

@abhijith7025
Copy link

@abhijith7025 abhijith7025 commented May 9, 2021

I think there is already a feature

files:

  • source:
    destination: c:\temp
    file_exists_behavior: OVERWRITE

read this out https://docs.aws.amazon.com/codedeploy/latest/userguide/reference-appspec-file-structure-files.html

@s-larbi
Copy link

@s-larbi s-larbi commented May 9, 2021

I have tried this before and it didn't work: "The deployment failed because a specified file already exists at this location: /var/www/html/website-folder/sub-folder/file-name.php"...

@0cool-f
Copy link

@0cool-f 0cool-f commented May 10, 2021

I think there is already a feature

files:

* source: 
  destination: c:\temp
  file_exists_behavior: OVERWRITE

read this out https://docs.aws.amazon.com/codedeploy/latest/userguide/reference-appspec-file-structure-files.html

mate you should always read before write something: this bug exists because this feature, which it seems you haven't tried, does not work, in fact it's a bug...do you really think that an issue could remain open and debated for years, waiting for a doc page that we all haven't read? 😶

@s-larbi
Copy link

@s-larbi s-larbi commented May 10, 2021

I hope this bug is fixed ASAP. It would be a great time saver.

@0cool-f
Copy link

@0cool-f 0cool-f commented May 10, 2021

I hope this bug is fixed ASAP. It would be a great time saver.

As Phil said, after the release cycle of 1.3.2 it will work.

Phil's team was fast, but "ASAP" is a relative concept, have you read the date of issue?

@philstrong
Copy link
Contributor

@philstrong philstrong commented May 10, 2021

1.3.2 is out in some regions e.g. us-east-1 but rolling out to the rest. We will update GitHub once all commercial regions are released which will be Thurs or Friday (5/13 or 5/14).
sa-east-1, ap-northeast-1, us-west-1, eu-west-1, ap-southeast-1 are also released

@quentinhayot
Copy link

@quentinhayot quentinhayot commented May 11, 2021

Jeff, Give this guy a cigar !

@philstrong
Copy link
Contributor

@philstrong philstrong commented May 24, 2021

Issue is fixed in 1.3.2. Any and all feedback is welcome.

@philstrong philstrong closed this May 24, 2021
@waldoj
Copy link

@waldoj waldoj commented May 24, 2021

Thank you, @philstrong, for getting this issue resolved. This will be a great little quality-of-life improvement for a lot of people. I'm grateful. 🙏🏻

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet