Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NEW Source Control module: git add/commit/push #57

Closed
wants to merge 21 commits into from
Closed
Show file tree
Hide file tree
Changes from 17 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
1 change: 1 addition & 0 deletions plugins/modules/git_commit.py
1 change: 1 addition & 0 deletions plugins/modules/git_push.py
Empty file.
135 changes: 135 additions & 0 deletions plugins/modules/source_control/git/git_commit.py
@@ -0,0 +1,135 @@
#!/usr/bin/python
# -*- coding: utf-8 -*-

# Copyright: (c) 2020, Federico Olivieri (lvrfrc87@gmail.com)
# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)

from __future__ import absolute_import, division, print_function
__metaclass__ = type


DOCUMENTATION = '''
---
module: git_commit
author:
- "Federico Olivieri (@Federico87)"
version_added: "2.10"
short_description: Perform git and and git commit operations.
lvrfrc87 marked this conversation as resolved.
Show resolved Hide resolved
description:
- Manage git add and git commit on local git repository.
lvrfrc87 marked this conversation as resolved.
Show resolved Hide resolved
options:
path:
description:
- Folder path where .git/ is located.
lvrfrc87 marked this conversation as resolved.
Show resolved Hide resolved
required: true
type: path
comment:
description:
- Git commit comment. Same as "git commit -m".
lvrfrc87 marked this conversation as resolved.
Show resolved Hide resolved
type: str
required: true
add:
description:
- list of files to be staged. Same as "git add ."
lvrfrc87 marked this conversation as resolved.
Show resolved Hide resolved
lvrfrc87 marked this conversation as resolved.
Show resolved Hide resolved
Asterisx values not accepted. i.e. "./*" or "*".
lvrfrc87 marked this conversation as resolved.
Show resolved Hide resolved
type: list
default: ["."]
elements: str
requirements:
- git>=2.10.0 (the command line tool)
'''

EXAMPLES = '''
- name: Add and commit 2 files.
lvrfrc87 marked this conversation as resolved.
Show resolved Hide resolved
community.general.git_commit:
path: /Users/federicoolivieri/git/git_test_module
comment: My amazing backup
add: ['test.txt', 'txt.test']

- name: Add all files using default and commit.
community.general.git_commit:
path: /Users/federicoolivieri/git/git_test_module
comment: My amazing backup
'''

RETURN = '''
output:
description: list of git cli command stdout
type: list
returned: always
sample: [
"[master 99830f4] Remove [ test.txt, tax.txt ]\n 4 files changed, 26 insertions(+)..."
]
'''

import os
from ansible.module_utils.basic import AnsibleModule


def git_commit(module):

commands = list()

add = module.params.get('add')
path = module.params.get('path')
comment = module.params.get('comment')

if add:
commands.append('git -C {path} add {add}'.format(
path=path,
add=' '.join(add),
))

if comment:
commands.append('git -C {path} commit -m "{comment}"'.format(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
commands.append('git -C {path} commit -m "{comment}"'.format(
commands.append('git -C {path} commit -m "{comment}"'.format(

Never ever use strings for commands! Use lists. This will also take care of all option quoting problems. (Your current version will break if someone uses " in a comment.)

(Note that module.run_command(cmd, ...) accepts lists for cmd.)

Copy link
Author

@lvrfrc87 lvrfrc87 Mar 30, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to make sure I understand properly:

Those strings are appended to a list that is then passed to run_command

    for cmd in git_commit(module):
        _rc, output, error = module.run_command(cmd, check_rc=False)

Is not that the same?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No. Consider module.run_command('echo "Hello world!"') vs. module.run_command(['echo', 'Hello world']). Here it is clear what are parameters 1 and 2 of the command, without having to parse it first with shell lexer rules.

Also assume that the commit message is x " hello " x. Then your call would result in the arguments -m, "x ", hello, " x" (four separate ones) passed to git. Using ['-m', 'x " hello " x'] would result in precisely two arguments being passed (with message exactly as provided by user).

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see what you mean know. It makes absolutely sense. I will fix that

Copy link
Author

@lvrfrc87 lvrfrc87 Mar 31, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding this bit, I have a problem to make it work.
This is the command: "cmds": ["git", "add", "1585649737984992100.txt 1585649738063034300.txt"]

When I run it against testhost I get {"changed": false, "cmd": "add", "msg": "[Errno 2] No such file or directory: b'add': b'add'", "rc": 2}

If I passe the command as string (using ' '.join()) the command is successful. I had a look around to other modules for git_config and I see they use a string for run_commands. I am bit confused :/

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just because other modules do bad things doesn't mean you should copy their behavior :) Using lists is a lot better than strings for run_command.

About that error: where exactly does that come from? Module output is not really helpful to understand what went wrong. What exactly did you pass to run_command, and what did run_command return?

path=path,
comment=comment,
))

return commands


def main():

argument_spec = dict(
path=dict(required=True, type="path"),
comment=dict(required=True),
add=dict(type="list", elements='str', default=["."]),
)

module = AnsibleModule(
argument_spec=argument_spec,
)

result_output = list()
result = dict(changed=False)

for cmd in git_commit(module):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, you should really get rid of this loop. It makes the module much harder to follow.

Just build and execute the commands sequentially.

_rc, output, error = module.run_command(cmd, check_rc=False)

if output:
if 'no changes added to commit' in output:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if output is localized?

For git commit, you should use the --porcelain option. No idea if something similar exists for git add.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I use --porcelain I am going to loose all the string outputs that I use for the output and error checking. rc in git are pretty useless as it pretty much always return 1 so I believe having strings to use as anchor for if statement is a good thing. If you have an idea how to implement a safe output error check using --porcelain please let me know. Here an example of what I mean:

(NaC) olivierif:NaC federicoolivieri$ git commit  -m "ciao"
On branch dev_infra
Your branch is ahead of 'origin/dev_infra' by 3 commits.
  (use "git push" to publish your local commits)

nothing to commit, working tree clean
(NaC) olivierif:NaC federicoolivieri$ git commit  -m "ciao"  --porcelain
(NaC) olivierif:NaC federicoolivieri$

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doing the output/error checks on string output is a terrible idea. This doesn't work at all.

Also please read man git-commit on what --porcelain does. In this case, empty output means "nothing to commit".

Copy link
Author

@lvrfrc87 lvrfrc87 Mar 31, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. What about for git add that does not support --porcelain. Is ok to use string output or do you have other ideas? I was thinking to use rc but I am not sure if distinguish between nothing to commit, working tree clean and fatal: pathspec 'file' did not match any files

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should run git status --porcelain=v1, like here: https://github.com/buildbot/buildbot/pull/5023/files#diff-380a2f2e80216ef42112b8b625db2516R753-R761

Then you know whether there's something to commit or not.

Parsing the normal text output of git is something you should NEVER rely on. There exist translated versions of CLI programs, and your code will do the wrong things when encountering such translated output.

module.fail_json(msg=output)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Failing when there's nothing to commit isn't useful in all cases. How about making that configurable?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. I would you call the argument? allow_empty ?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, wait, I think it is better to distinguish between three behaviors:

  • fail if nothing to commit;
  • don't commit if there's nothing to commit;
  • add empty commit if there's nothing to commit (git's --allow-empty option).

Maybe call the option empty_commit and make it an enum with values fail, skip and empty_commit. Or something similar.

(Now it also reminds me where I've seen this before: https://docs.buildbot.net/current/manual/configuration/buildsteps.html#gitcommit has an option emptyCommits. I remembered that I have actually added it because I needed it once :D Here's the PR: buildbot/buildbot#5023 that might help you with implementing this.)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's ok. Good suggestion. That line you indicated though checks if there are not files to add. The commit check is the line below:

            elif 'nothing to commit, working tree clean' in output:
                module.fail_json(msg=output)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Developing the empty_commit=skip I realize that is not really useful in my opinion. Basically, if I am not mistaken skip mode will allow the module just to stage file and personally I cannot see a use case where you want use a specific ansible module just to add files.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

empty_commit=skip means: if there's nothing to add (stage), don't commit.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, empty_commit=skip would be conditional for git add command while empty_commit=allow|fail would be conditional for git commit. Is that correct? If that is the case, would not be confusing that one argument affect the behavior of 2 different thighs?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It all determines on what git commit does. Either it is potentially not run (skip), it gets the --empty-commit flag (allow), or it is run as usual (fail).

elif 'nothing to commit, working tree clean' in output:
module.fail_json(msg=output)
else:
result_output.append(output)
result.update(changed=True)

if error:
if 'error:' in error:
module.fail_json(msg=error)
elif 'fatal:' in error:
module.fail_json(msg=error)
else:
result_output.append(error)
result.update(changed=True)

if result_output:
result.update(output=result_output)

module.exit_json(**result)


if __name__ == "__main__":
main()
235 changes: 235 additions & 0 deletions plugins/modules/source_control/git/git_push.py
@@ -0,0 +1,235 @@
#!/usr/bin/python
# -*- coding: utf-8 -*-

# Copyright: (c) 2020, Federico Olivieri (lvrfrc87@gmail.com)
# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)

from __future__ import absolute_import, division, print_function
__metaclass__ = type


DOCUMENTATION = '''
---
module: git_push
author:
- "Federico Olivieri (@Federico87)"
version_added: "2.10"
short_description: Perform git push operations.
description:
- Manage git push on local or remote git repository.
options:
path:
description:
- Folder path where .git/ is located.
required: true
type: path
user:
description:
- Git username for https operations.
type: str
token:
description:
- Git API token for https operations.
type: str
branch:
description:
- Git branch where perform git push.
required: True
type: str
push_option:
description:
- Git push options. Same as "git --push-option=option".
type: str
mode:
description:
- Git operations are performend eithr over ssh, https or local.
Same as "git@git..." or "https://user:token@git..." or "git init --bare"
choices: ['ssh', 'https', 'local']
default: ssh
type: str
url:
description:
- Git repo URL.
required: True
type: str
requirements:
- git>=2.10.0 (the command line tool)
'''

EXAMPLES = '''

- name: Push changes via HTTPs.
community.general.git_push:
path: /Users/federicoolivieri/git/git_test_module
user: Federico87
token: m1Ap!T0k3n!!!
branch: master
mode: https
url: https://gitlab.com/networkAutomation/git_test_module

- name: Push changes via SSH.
community.general.git_push:
path: /Users/federicoolivieri/git/git_test_module
branch: master
mode: ssh
url: https://gitlab.com/networkAutomation/git_test_module

- name: Push changes on local repo.
community.general.git_push:
path: /Users/federicoolivieri/git/git_test_module
comment: My amazing backup
branch: master
url: /Users/federicoolivieri/git/local_repo
'''

RETURN = '''
output:
description: list of git cli commands stdout
type: list
returned: always
sample: [
"To https://gitlab.com/networkAutomation/git_test_module.git\n 372db19..99830f4 master -> master\n"
]
'''

import os
from ansible.module_utils.basic import AnsibleModule


def git_push(module):

commands = list()

path = module.params.get('path')
url = module.params.get('url')
user = module.params.get('user')
token = module.params.get('token')
branch = module.params.get('branch')
push_option = module.params.get('push_option')
mode = module.params.get('mode')

def https(path, user, token, url, branch, push_option):
if url.startswith('https://'):
remote_add = 'git -C {path} remote set-url origin https://{user}:{token}@{url}'.format(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also here: use lists for commands, not strings.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And add --porcelain.

path=path,
url=url[8:],
user=user,
token=token,
)
cmd = 'git -C {path} push origin {branch}'.format(
path=path,
branch=branch,
)

if push_option:
index = cmd.find('origin')
return [remote_add, cmd[:index] + '--push-option={option} '.format(option=push_option) + cmd[index:]]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will break if url does not starts with https://, since then remote_add has not been defined.

Copy link
Author

@lvrfrc87 lvrfrc87 Mar 30, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would be just mater of indent that block under if url.startswith('https://'): Would that work for you?

    def https(path, user, token, url, branch, push_option):
        if url.startswith('https://'):
            remote_add = 'git -C {path} remote set-url origin https://{user}:{token}@{url}'.format(
                path=path,
                url=url[8:],
                user=user,
                token=token,
            )
            cmd = 'git -C {path} push origin {branch}'.format(
                path=path,
                branch=branch,
            )

            if push_option:
                index = cmd.find('origin')
                return [remote_add, cmd[:index] + '--push-option={option} '.format(option=push_option) + cmd[index:]]

            if not push_option:
                return [remote_add, cmd]

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should rather raise an error when url does not start with https://.


if not push_option:
return [remote_add, cmd]

if mode == 'local':
if 'https' in url or 'ssh' in url:
module.fail_json(msg='SSH or HTTPS mode selected but repo is LOCAL')

remote_add = 'git -C {path} remote set-url origin {url}'.format(
path=path,
url=url
)
cmd = "git -C {path} push origin {branch}".format(
path=path,
branch=branch
)

if push_option:
index = cmd.find('origin')
return [remote_add, cmd[:index] + '--push-option={option} '.format(option=push_option) + cmd[index:]]

if not push_option:
return [remote_add, cmd]

if mode == 'https':
for cmd in https(path, user, token, url, branch, push_option):
commands.append(cmd)

if mode == 'ssh':
if 'https' in url:
module.fail_json(msg='SSH mode selected but HTTPS URL provided')

remote_add = 'git -C {path} remote set-url origin {url}'.format(
path=path,
url=url,
)
cmd = 'git -C {path} push origin {branch}'.format(
path=path,
branch=branch
)
commands.append(remote_add)

if push_option:
index = cmd.find('origin')
commands.append(cmd[:index] + '--push-option={option} '.format(option=push_option) + cmd[index:])

if not push_option:
commands.append(cmd)

return commands


def main():

argument_spec = dict(
path=dict(required=True, type="path"),
user=dict(),
token=dict(),
branch=dict(required=True),
push_option=dict(),
mode=dict(choices=["ssh", "https", "local"], default='ssh'),
url=dict(required=True),
)

required_if = [
("mode", "https", ["user", "token"]),
]

module = AnsibleModule(
argument_spec=argument_spec,
required_if=required_if,
)

result = dict(changed=False)

result_output = list()

for cmd in git_push(module):
_rc, output, error = module.run_command(cmd, check_rc=False)

if output:
if 'no changes added to commit' in output:
module.fail_json(msg=output)
elif 'nothing to commit, working tree clean' in output:
module.fail_json(msg=output)
else:
result_output.append(output)
result.update(changed=True)

if error:
if 'error:' in error:
module.fail_json(msg=error)
elif 'fatal:' in error:
module.fail_json(msg=error)
elif 'Everything up-to-date' in error:
result_output.append(error)
result.update(changed=True)
else:
result_output.append(error)
result.update(changed=True)

if result_output:
result.update(output=result_output)

module.exit_json(**result)


if __name__ == "__main__":
main()
2 changes: 2 additions & 0 deletions tests/integration/targets/git_commit/aliases
@@ -0,0 +1,2 @@
shippable/posix/group4
skip/aix
3 changes: 3 additions & 0 deletions tests/integration/targets/git_commit/meta/main.yml
@@ -0,0 +1,3 @@
dependencies:
- prepare_tests
lvrfrc87 marked this conversation as resolved.
Show resolved Hide resolved
- setup_pkg_mgr