Use .ebignore file when deploying to Elastic Beanstalk

[petridw]

Would it be possible to have the Elastic Beanstalk provider use the .ebignore file if it is present when creating the .zip to deploy to Elastic Beanstalk, rather than always using .gitignore?

[BanzaiMan]

http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/eb-cli3-configuration.html#eb-cli3-ebignore

Seems reasonable.

[aripalo]

Yeah, using .gitignore instead of .ebignore (as is the AWS default when .ebignore exists) messes things up: Basically now any build time files - which are git ignored and kept out of vcs - won't get deployed.

In short: the Elastic Beanstalk deployment is broken.

[aripalo]

As .ebignore uses the same syntax as .gitignore, I decided to google for a gem that would handle ignore rule vs. file path matching: The only one I found was https://github.com/nathankleyn/gitignore.rb. Which I didn't manage to get installed, it is a wrapper to Rust code, and its Github repository doesn't seem active / popular.

So as an exercise I tried making one myself. It seems to work, but I'll have to test it more with various diffent .ebignore files. If anyone is interested, some help with testing would be much appreciated.

The code can be found here: https://gist.github.com/aripalo/b544370b124b23de8c07f81b38e6721f

When I'm "done", I'll wrap it as a gem, publish it, implement it within the Elastic Beanstalk provider files_to_pack-function and make a pull request.

… unless of course if anyone comes out with a better solution! 🙂

[jdub]

This is a pretty serious security issue for anyone who moves their deployment process to Travis without realising this doesn't work. .ebignore is where you put all the stuff that needs to be in your git repo, but should not be deployed.

[jdub]

Here's a cheesy workaround. Please be careful: It will only work with relatively simple .ebignore files.

  # - Deal with lack of .ebignore support in Travis CI's dpl
  #   https://github.com/travis-ci/dpl/issues/411
  # - Use `git rm` so the removed files don't appear in `git ls-files`
  #   https://github.com/travis-ci/dpl/blob/master/lib/dpl/provider/elastic_beanstalk.rb#L106  
  - grep ^/ .ebignore | while read IGNORE; do git rm -rf ".${IGNORE}"; done
  - grep -v ^/ .ebignore | while read IGNORE; do find . -name "${IGNORE}" -print0 | xargs git rm -rf; done
  - git rm -f .ebignore

[jdub]

I discovered that git ls-files accepts a --exclude-from= parameter, which is supposed support .gitignore style exclude patterns. With that, we could just add --exclude-from=.ebignore to resolve this.

Unfortunately, it still lists files that ought to be matched by my .ebignore file. Annoying!

[jdub]

Ah, the description for -x explains why: Skip untracked files matching pattern.

git ls-files --ignored --exclude-from=.ebignore shows just the ignored files, simplifying my .travis.yml snippet above to:

- git ls-files --ignored --exclude-from=.ebignore -z | xargs -0 git rm -f
- git rm -f .ebignore

[oppodeldoc]

I was able to solve this in maybe a hacky fashion. I wanted to run all my builds and tests in Travis and then send over the dist folder with my production build files, which is obviously .gitignored.

My solution was just to force add the dist folder, then commit it to the local repository in Travis. To do that, I had to set some configs so that it didn't get mad. So my before_deploy hook in .travis.ymllooks like:

before_deploy:
- git config user.email "you@example.com"
- git config user.name "Your Name"
- git add --force dist # Whatever you want to include in your EB deploy
- git commit -m 'adds dist folder'

It's working now. Hope this helps someone. This is technically expected behavior but I wasn't clear on how the EB deployments work behind the scenes, I'll see if I can submit a PR in the docs to clarify.

[jdub]

@oppodeldoc Luckily, you don't have to commit, just stage your changes. Makes things a bit simpler. :-)

[oppodeldoc]

Hi, thanks @jdub but I tried that and the files still weren't added, here's the output from the Travis console:

Deploying application
# HEAD detached at 9dbea66
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#	new file:   dist/0.discover.69df7ebc67cb63832089.js
#	new file:   dist/0e34d2fd02afa421b79480e3b5099d7f.jpg
#	new file:   dist/1.program.9fb7f84d3f9570968764.js
#	new file:   dist/2.listen.f2834f64045486cad4fc.js
#	new file:   dist/38af2ccdffd24157fdc0c8e20b47e7ff.jpg
#	new file:   dist/451e850bc7c65295e44133a0efc5357d.jpg
#	new file:   dist/5f98987fc42eb998a4130156676baa83.jpg
#	new file:   dist/65e363411684e755012e5ed7a931b1bc.jpg
#	new file:   dist/8c220465c587f073baa46aad2e7da9f2.jpg
#	new file:   dist/app.96cb54ceae7499b65e97.js
#	new file:   dist/app.bd1a3a07a30f483cf7e5bf79008b9917.css
#	new file:   dist/b9f792450e92304e19923a8ba2cd7526.jpg
#	new file:   dist/eacad6d50a1905465149c4a472d3945f.png
#	new file:   dist/f06ec141d257ba4518a0fab22ba463bc.jpg
#	new file:   dist/f8f1b41c9ac32e6e49992a3015024ae1.jpg
#	new file:   dist/favicon.ico
#	new file:   dist/humans.txt
#	new file:   dist/index.html
#	new file:   dist/robots.txt
#	new file:   dist/vendor.85fab527040f3b9dda61.js
#
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#	travis-9dbea661dab0636d80b89871df19787d05c0213b-1497391823.zip
#	travis_phantomjs/
#
Dropped refs/stash@{0} (602f682d7123483950d898f045b914e46bf7ae54)
Done. Your build exited with 0.

So, it looks like it deploys them but the entire dist file is missing from my deployment on EB, maybe I'm missing something here?

Just to check, I pulled down the zip file listed from S3 and unzipped it. The dist file was missing from it.

[stale]

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically closing the issue. This is often because the request was already solved in some way and it just wasn't updated or it's no longer applicable. If that's not the case, please do feel free to either reopen this issue or open a new one. We'll gladly take a look again! You can read more here: https://blog.travis-ci.com/2018-03-09-closing-old-issues

[jdub]

This is definitely still an ongoing issue.

[stale]

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically closing the issue. This is often because the request was already solved in some way and it just wasn't updated or it's no longer applicable. If that's not the case, please do feel free to either reopen this issue or open a new one. We'll gladly take a look again! You can read more here: https://blog.travis-ci.com/2018-03-09-closing-old-issues

[jdub]

This is definitely still an ongoing issue.

[stale]

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically closing the issue. This is often because the request was already solved in some way and it just wasn't updated or it's no longer applicable. If that's not the case, please do feel free to either reopen this issue or open a new one. We'll gladly take a look again! You can read more here: https://blog.travis-ci.com/2018-03-09-closing-old-issues

[jdub]

Rude, stalebot.

[rmacqueen]

We will not go gentle into that good night

[BanzaiMan]

Please test #885 with the following configuration and report results:

deploy:
  provider: elasticbeanstalk
  edge:
    branch: ebignore
  ⋮ # rest

1. If .ebignore exists, read that file, and that file only, to choose files to include.
2. If .ebignore does not exist, but .gitignore does, then use the standard rules to choose files. (See https://git-scm.com/docs/git-ls-files#_exclude_patterns)

The latter is a departure from the current behavior, and may require some care.

[BanzaiMan]

Anyone interested in this? I'd like to add it to 1.10.5.

[chalupagrande]

@BanzaiMan : Didn't work for me..

I was able to get it to work using @oppodeldoc 's work around. Incidentally. I didn't need to commit the changes. Just stage them as @jdub said.

Here is my travis.yml:

language: node_js
jobs:
  include:
    - stage: build & deploy
      before_script: bash scripts/travisPrepForDeployment.sh
      script: yarn build
      deploy:
        provider: elasticbeanstalk
        edge:
          branch: ebignore
        access_key_id: [my-access-key-id]
        secret_access_key:
          secure: [my-secret-access-key]
        region: us-east-2
        app: aws-test
        env: aws-test-env
        bucket_name: [my_bucket_name]
        bucket_path: 'aws-test'
        skip_cleanup: true

My .ebignore:

/node_modules
/coverage
.DS_Store

npm-debug.log*
yarn-debug.log*
yarn-error.log*

my .gitignore (just from create-react-app):

# ...

# dependencies
/node_modules

# production
/build

# ...

My build didn't make it into the s3 zip folder.

Here are the logs from the deployment step:
travis.log

[BanzaiMan]

@ninth-mind Is this all of the logs?

[chalupagrande]

@BanzaiMan All of the logs after my build step, yes. The build exited with a 0
Here they are again with a little more info:
travis.log

[BanzaiMan]

@ninth-mind Did you have skip_cleanup: true? Do you have a build log URL for this?

[svenfuchs]

Here's a new PR superseding the previous one #1057. This stops using git ls-files, and uses a Ruby port of Python's PathSpec (which is what the EB CLI uses for this filtering).

This can be tested with:

deploy:
  provider: elasticbeanstalk
  edge:
    branch: master
  ⋮ 

Note that this targets the master branch, which has the new version v2, which is tested, but not yet released. So there potentially might be other issues. If so we'd love to get feedback on that.

[svenfuchs]

I am going to close this issue here as I believe this is addressed in master (dpl v2). Please feel free to re-open if needed.

[mvakula]

@svenfuchs what was the solution? I still have the issue when using travis in production (http://travis-ci.com) Or when will it be released?

[svenfuchs]

@mvakula it's addressed in the v2 prerelease version, see https://blog.travis-ci.com/2019-08-27-deployment-tooling-dpl-v2-preview-release. you can opt in via:

deploy:
  - provider: [your-provider]
    edge: true
    ...

Does that work for you?

[mvakula]

@svenfuchs yes, thanks!