New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

macOS's grep is not compatible gnu grep #32

Open
n0ts opened this Issue Aug 2, 2016 · 12 comments

Comments

Projects
None yet
6 participants
@n0ts

n0ts commented Aug 2, 2016

macOS's grep (BSD grep 2.5.1-FreeBSD) is not compatible gnu grep.
So, commit-msg hook is always ERROR.

git secrets --commit_msg_hook -- foo.txt
test1.txt:1:test1

[ERROR] Matched one or more prohibited patterns

Possible mitigations:
- Mark false positives as allowed using: git config --add secrets.allowed ...
- Mark false positives as allowed by adding regular expressions to .gitallowed at repository's root directory
- List your configured patterns: git config --get-all secrets.patterns
- List your configured allowed patterns: git config --get-all secrets.allowed
- List your configured allowed patterns in .gitallowed at repository's root directory
- Use --no-verify if this is a one-time false positive

@n0ts n0ts changed the title from macOS's grep is not compatible to macOS's grep is not compatible gnu grep Aug 2, 2016

@mtdowling

This comment has been minimized.

Show comment
Hide comment
@mtdowling

mtdowling Aug 11, 2016

Contributor

The tests for git-secrets works both on my mac and on Travis, so I haven't run into this issue.

Can you give an example of a regex that doesn't work on both?

Contributor

mtdowling commented Aug 11, 2016

The tests for git-secrets works both on my mac and on Travis, so I haven't run into this issue.

Can you give an example of a regex that doesn't work on both?

@n0ts

This comment has been minimized.

Show comment
Hide comment
@n0ts

n0ts Sep 30, 2016

@mtdowling Thanks. Sorry for late reply. Hmm, I tried more times, I also working...

n0ts commented Sep 30, 2016

@mtdowling Thanks. Sorry for late reply. Hmm, I tried more times, I also working...

@n0ts n0ts closed this Sep 30, 2016

@n0ts n0ts reopened this Sep 30, 2016

@n0ts

This comment has been minimized.

Show comment
Hide comment
@n0ts

n0ts Sep 30, 2016

@mtdowling I use gnu grep 2.25 via homebrew (brew install --with-default-names grep).

$ grep --version
grep (GNU grep) 2.25
Packaged by Homebrew
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Mike Haertel and others, see <http://git.sv.gnu.org/cgit/grep.git/tree/AUTHORS>.

My gitconfig grep and secret is below.

[pager]
  grep = false

[secrets]
  providers = git secrets --aws-provider
  patterns = [A-Z0-9]{20}
  patterns = (\"|')?(AWS|aws|Aws)?_?(SECRET|secret|Secret)?_?(ACCESS|access|Access)?_?(KEY|key|Key)(\"|')?\\s*(:|=>|=)\\s*(\"|')?[A-Za-z0-9/\\+=]{40}(\"|')?
  patterns = (\"|')?(AWS|aws|Aws)?_?(ACCOUNT|account|Account)_?(ID|id|Id)?(\"|')?\\s*(:|=>|=)\\s*(\"|')?[0-9]{4}\\-?[0-9]{4}\\-?[0-9]{4}(\"|')?
  allowed = AKIAIOSFODNN7EXAMPLE
  allowed = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

So I create new files into the git repository.
When I commit new files got a below error.

$ git init .
$ touch hoge_{1..10}
$ git add hoge_*
$ git commit -m "test" .
fatal: ambiguous argument 'hoge_1 hoge_10 hoge_2 hoge_3 hoge_4 hoge_5 hoge_6 hoge_7 hoge_8 hoge_9': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'

I think that git-secret regular patterns is not compatible gnu grep.

$ git grep -nwHEI --cached "[A-Z0-9]{20}|("|')?(AWS|aws|Aws)?_?(SECRET|secret|Secret)?_?(ACCESS|access|Access)?_?(KEY|key|Key)("|')?\s*(:|=>|=)\s*("|')?[A-Za-z0-9/\+=]{40}("|')?|("|')?(AWS|aws|Aws)?_?(ACCOUNT|account|Account)_?(ID|id|Id)?("|')?\s*(:|=>|=)\s*("|')?[0-9]{4}\-?[0-9]{4}\-?[0-9]{4}("|')?|AKIAJYEVZ2UZKEDU3B2A|6U4Iwm/0u9aNPyIFA\+KYRkPZi42LDQMhAw9zlPNn" hoge_1 hoge_10 hoge_2 hoge_3 hoge_4 hoge_5 hoge_6 hoge_7 hoge_8 hoge_9
bash: syntax error near unexpected token `)'

n0ts commented Sep 30, 2016

@mtdowling I use gnu grep 2.25 via homebrew (brew install --with-default-names grep).

$ grep --version
grep (GNU grep) 2.25
Packaged by Homebrew
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Mike Haertel and others, see <http://git.sv.gnu.org/cgit/grep.git/tree/AUTHORS>.

My gitconfig grep and secret is below.

[pager]
  grep = false

[secrets]
  providers = git secrets --aws-provider
  patterns = [A-Z0-9]{20}
  patterns = (\"|')?(AWS|aws|Aws)?_?(SECRET|secret|Secret)?_?(ACCESS|access|Access)?_?(KEY|key|Key)(\"|')?\\s*(:|=>|=)\\s*(\"|')?[A-Za-z0-9/\\+=]{40}(\"|')?
  patterns = (\"|')?(AWS|aws|Aws)?_?(ACCOUNT|account|Account)_?(ID|id|Id)?(\"|')?\\s*(:|=>|=)\\s*(\"|')?[0-9]{4}\\-?[0-9]{4}\\-?[0-9]{4}(\"|')?
  allowed = AKIAIOSFODNN7EXAMPLE
  allowed = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

So I create new files into the git repository.
When I commit new files got a below error.

$ git init .
$ touch hoge_{1..10}
$ git add hoge_*
$ git commit -m "test" .
fatal: ambiguous argument 'hoge_1 hoge_10 hoge_2 hoge_3 hoge_4 hoge_5 hoge_6 hoge_7 hoge_8 hoge_9': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'

I think that git-secret regular patterns is not compatible gnu grep.

$ git grep -nwHEI --cached "[A-Z0-9]{20}|("|')?(AWS|aws|Aws)?_?(SECRET|secret|Secret)?_?(ACCESS|access|Access)?_?(KEY|key|Key)("|')?\s*(:|=>|=)\s*("|')?[A-Za-z0-9/\+=]{40}("|')?|("|')?(AWS|aws|Aws)?_?(ACCOUNT|account|Account)_?(ID|id|Id)?("|')?\s*(:|=>|=)\s*("|')?[0-9]{4}\-?[0-9]{4}\-?[0-9]{4}("|')?|AKIAJYEVZ2UZKEDU3B2A|6U4Iwm/0u9aNPyIFA\+KYRkPZi42LDQMhAw9zlPNn" hoge_1 hoge_10 hoge_2 hoge_3 hoge_4 hoge_5 hoge_6 hoge_7 hoge_8 hoge_9
bash: syntax error near unexpected token `)'
@n0ts

This comment has been minimized.

Show comment
Hide comment
@n0ts

n0ts Oct 26, 2016

@mtdowling What is status this issue? Thanks.

n0ts commented Oct 26, 2016

@mtdowling What is status this issue? Thanks.

@mtdowling

This comment has been minimized.

Show comment
Hide comment
@mtdowling

mtdowling Oct 26, 2016

Contributor

What version of git-secrets are you using? Can you try updating to the latest version? The ambiguous argument error looks unrelated to the regular expression that's being used.

As for the second, bash syntax error, that looks to be caused by not escaping the regex properly.

Contributor

mtdowling commented Oct 26, 2016

What version of git-secrets are you using? Can you try updating to the latest version? The ambiguous argument error looks unrelated to the regular expression that's being used.

As for the second, bash syntax error, that looks to be caused by not escaping the regex properly.

@n0ts

This comment has been minimized.

Show comment
Hide comment
@n0ts

n0ts Oct 27, 2016

I use git-scretes 1.2.1.

n0ts commented Oct 27, 2016

I use git-scretes 1.2.1.

@mtdowling

This comment has been minimized.

Show comment
Hide comment
@mtdowling

mtdowling Nov 1, 2016

Contributor

I've been unable to reproduce this so far. What version of git are you using?

Contributor

mtdowling commented Nov 1, 2016

I've been unable to reproduce this so far. What version of git are you using?

@n0ts

This comment has been minimized.

Show comment
Hide comment
@n0ts

n0ts Nov 2, 2016

Thanks, Now I use git 2.10.1,

n0ts commented Nov 2, 2016

Thanks, Now I use git 2.10.1,

@ricoli

This comment has been minimized.

Show comment
Hide comment
@ricoli

ricoli Jun 20, 2017

any progress on this? Also having same issue.

ricoli commented Jun 20, 2017

any progress on this? Also having same issue.

@f440

This comment has been minimized.

Show comment
Hide comment
@f440

f440 Sep 18, 2017

Steps to reproduce...

  1. Create Dockerfile:
FROM ubuntu:17.04
# Ubuntu 16.10 (git 2.9.3 & grep 2.25)  : OK
# Ubuntu 17.04 (git 2.11.0 & grep 2.27) : NG

RUN apt-get update \
  && apt-get install -y git

RUN apt-get install -y curl
RUN curl -L -o /tmp/git-secrets \
  https://raw.githubusercontent.com/awslabs/git-secrets/1.2.1/git-secrets
RUN install /tmp/git-secrets /usr/local/bin/

RUN git config --global user.email "you@example.com" \
  && git config --global user.name "Your Name"

RUN mkdir /app
WORKDIR /app
RUN git --version ; grep --version | grep '^grep'
RUN git init
RUN git secrets --install && git secrets --register-aws

RUN touch a b c
RUN git add .
RUN git commit -m "Initial commit"
  1. docker build .

f440 commented Sep 18, 2017

Steps to reproduce...

  1. Create Dockerfile:
FROM ubuntu:17.04
# Ubuntu 16.10 (git 2.9.3 & grep 2.25)  : OK
# Ubuntu 17.04 (git 2.11.0 & grep 2.27) : NG

RUN apt-get update \
  && apt-get install -y git

RUN apt-get install -y curl
RUN curl -L -o /tmp/git-secrets \
  https://raw.githubusercontent.com/awslabs/git-secrets/1.2.1/git-secrets
RUN install /tmp/git-secrets /usr/local/bin/

RUN git config --global user.email "you@example.com" \
  && git config --global user.name "Your Name"

RUN mkdir /app
WORKDIR /app
RUN git --version ; grep --version | grep '^grep'
RUN git init
RUN git secrets --install && git secrets --register-aws

RUN touch a b c
RUN git add .
RUN git commit -m "Initial commit"
  1. docker build .
@profburke

This comment has been minimized.

Show comment
Hide comment
@profburke

profburke Dec 14, 2017

I am also seeing this problem.

Note that I can commit one file at a time. But if I try to commit several files, I get an error message like

fatal: ambiguous argument 'file1 file2': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'

macOS 10.12.6
git-secrets 1.2.1
git 2.15.1

I am also seeing this problem.

Note that I can commit one file at a time. But if I try to commit several files, I get an error message like

fatal: ambiguous argument 'file1 file2': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'

macOS 10.12.6
git-secrets 1.2.1
git 2.15.1

@denihidayat157

This comment has been minimized.

Show comment
Hide comment

Aktifkan

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment