Can't clone repo on MacOS

[lindycoder]

Hi!

For some weird reason when cloning the repo on a mac (tested with 10.11 and 10.13) the file pyt/trigger_definitions/flask_trigger_words.pyt won't be written.

here's an example:

} /tmp$ git clone https://github.com/python-security/pyt.git
Cloning into 'pyt'...
remote: Counting objects: 5740, done.
remote: Total 5740 (delta 0), reused 0 (delta 0), pack-reused 5740
Receiving objects: 100% (5740/5740), 2.62 MiB | 3.75 MiB/s, done.
Resolving deltas: 100% (3916/3916), done.
Checking connectivity... done.
} /tmp$ cd pyt/
} /tmp/pyt$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

	deleted:    pyt/trigger_definitions/flask_trigger_words.pyt

no changes added to commit (use "git add" and/or "git commit -a")

Even copy pasting the content in a file result in the file not existing. Tried with default terminal, iterm2 and intellij's terminal, all the same so musn't be the terminal.

Doing some try/fail we suspect that the faulty line is subprocess.call( but doing a hexdump of the file (on a xenial box) doesn't show much...

root@web1:~/pyt/pyt/trigger_definitions# cat flask_trigger_words.pyt | hexdump -C
00000000  73 6f 75 72 63 65 73 3a  0a 67 65 74 28 0a 2e 64  |sources:.get(..d|
00000010  61 74 61 0a 66 6f 72 6d  5b 0a 66 6f 72 6d 28 0a  |ata.form[.form(.|
00000020  4d 61 72 6b 75 70 28 0a  63 6f 6f 6b 69 65 73 5b  |Markup(.cookies[|
00000030  0a 66 69 6c 65 73 5b 0a  53 51 4c 41 6c 63 68 65  |.files[.SQLAlche|
00000040  6d 79 0a 0a 73 69 6e 6b  73 3a 0a 72 65 70 6c 61  |my..sinks:.repla|
00000050  63 65 28 20 2d 3e 20 65  73 63 61 70 65 0a 73 65  |ce( -> escape.se|
00000060  6e 64 5f 66 69 6c 65 28  20 2d 3e 20 27 2e 2e 27  |nd_file( -> '..'|
00000070  2c 20 27 2e 2e 27 20 69  6e 0a 65 78 65 63 75 74  |, '..' in.execut|
00000080  65 28 0a 73 79 73 74 65  6d 28 0a 66 69 6c 74 65  |e(.system(.filte|
00000090  72 28 0a 73 75 62 70 72  6f 63 65 73 73 2e 63 61  |r(.subprocess.ca|
000000a0  6c 6c 28 0a 72 65 6e 64  65 72 5f 74 65 6d 70 6c  |ll(.render_templ|
000000b0  61 74 65 28 0a 73 65 74  5f 63 6f 6f 6b 69 65 28  |ate(.set_cookie(|
000000c0  0a 72 65 64 69 72 65 63  74 28 0a 75 72 6c 5f 66  |.redirect(.url_f|
000000d0  6f 72 28 0a 66 6c 61 73  68 28 0a 6a 73 6f 6e 69  |or(.flash(.jsoni|
000000e0  66 79 28                                          |fy(|
000000e3

The result of this is the tool can't seem to run on mac since this file is not available, fails with

Traceback (most recent call last):
  File ".../bin/pyt", line 11, in <module>
    load_entry_point('pyt==1.0.0a20', 'console_scripts', 'pyt')()
  File ".../lib/python3.5/site-packages/pyt-1.0.0a20-py3.5.egg/pyt/__main__.py", line 247, in main
    args.trim_reassigned_in)
  File ".../lib/python3.5/site-packages/pyt-1.0.0a20-py3.5.egg/pyt/vulnerabilities.py", line 394, in find_vulnerabilities
    definitions = parse(trigger_word_file)
  File ".../lib/python3.5/site-packages/pyt-1.0.0a20-py3.5.egg/pyt/trigger_definitions_parser.py", line 48, in parse
    with open(trigger_word_file, 'r') as fd:
FileNotFoundError: [Errno 2] No such file or directory: '.../lib/python3.5/site-packages/pyt-1.0.0a20-py3.5.egg/pyt/trigger_definitions/flask_trigger_words.pyt'

Does that ring any bell?

[KevinHock]

I can definitely take a look at this around December 23rd, thanks for bringing this to our attention. 👍

[Thalmann]

A friend of mine tried to clone the repo using MacOS 10.13.2
and git 2.11.0 with the folowing outcome: https://gist.github.com/con-/c031049572531f5a2f59547177f0478b

So i think we need more details or it might be a configuration on your machine :)

[lindycoder]

Well we had this on 2 separate computers here so it's probably not my machine, but could be my company's machines :-/

I am really not sure what more details i can give... but if nobody else can reproduce this, i'll assume we have a weird in house problem

[Thalmann]

I agree with you that this is a weird issue.

Somehow we need some more info. One way to get more is using verbose cloning over https to see if some packages are not sent.

GIT_CURL_VERBOSE=1 git clone https://github.com/python-security/pyt.git

Is it possible for you to try on a private network, not your companies?
Is it possible to try it on a mac not from your company? (We kind of confirmed that already worked earlier so idk how necessary this is..)

I am just putting some thoughts out there so we can find a solution :)

[Thalmann]

Can try to clone into another folder than the /tmp folder? Would seem weird if that is the error but just to rule that option out :)

[KevinHock]

Hi, @lindycoder hope all is well, are you still experiencing this issue?

[lindycoder]

Hey, sorry i forgot about this....

I've got a brand new Mac running 10.12.6 and the issue is still happening.

I tried different shells : iTerm2 vs. Terminal vs. Intellij's built in Terminal, all same

I tried outside my company's network, stil the same.

for @Thalmann , only my example was in /tmp the problem is present in other folders.

So frankly it must be my company's management tool or the way the disk is encrypted (i remember i had to enter a password for the disk encryption).

So there, since nobody else seems to have come to whine about it it must be our problem, i guess we can close this.

Thanks for looking into it!