Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make file state=absent respect recurse=false for previous state=directory #52656

Open
wants to merge 23 commits into
base: devel
from

Conversation

@ewarnke
Copy link

ewarnke commented Feb 20, 2019

SUMMARY

Make file state=absent respect recuse=false for previos state=directory

Fixes #52653

ISSUE TYPE
  • Bugfix Pull Request

Sorry about the multiple pulls, not a frequent contributor.

@ewarnke

This comment has been minimized.

Copy link
Author

ewarnke commented Feb 20, 2019

Why is shippable using the wrong code branch!?

@ewarnke

This comment has been minimized.

Copy link
Author

ewarnke commented Feb 20, 2019

Tests need to be updated to set recurse=true when they want to delete non-empty directory

@ansibot

This comment was marked as resolved.

Copy link
Contributor

ansibot commented Feb 20, 2019

@ewarnke this PR contains the following merge commits:

Please rebase your branch to remove these commits.

click here for bot help

@dagwieers
Copy link
Member

dagwieers left a comment

LGTM, but this changes behaviour.

Show resolved Hide resolved lib/ansible/modules/files/file.py Outdated
Show resolved Hide resolved lib/ansible/modules/files/file.py Outdated

@dagwieers dagwieers changed the title Make file state=absent respect recuse=false for previos state=directory Make file state=absent respect recurse=false for previous state=directory Feb 20, 2019

@ansibot

This comment was marked as resolved.

Copy link
Contributor

ansibot commented Feb 20, 2019

@ewarnke this PR contains the following merge commits:

Please rebase your branch to remove these commits.

click here for bot help

@ansibot ansibot removed the needs_triage label Feb 20, 2019

@ewarnke ewarnke force-pushed the ewarnke:patch-2 branch from 168b355 to 92d0ca1 Feb 20, 2019

@ansibot

This comment was marked as resolved.

Copy link
Contributor

ansibot commented Feb 20, 2019

The test ansible-test sanity --test pylint [explain] failed with 2 errors:

lib/ansible/modules/files/file.py:265:52: bad-whitespace No space allowed after bracket     if params['recurse'] and params['state'] not in [  'directory', 'absent' ]:                                                     ^
lib/ansible/modules/files/file.py:265:77: bad-whitespace No space allowed before bracket     if params['recurse'] and params['state'] not in [  'directory', 'absent' ]:                                                                              ^

The test ansible-test sanity --test pep8 [explain] failed with 3 errors:

lib/ansible/modules/files/file.py:35:91: W291 trailing whitespace
lib/ansible/modules/files/file.py:265:54: E201 whitespace after '['
lib/ansible/modules/files/file.py:265:77: E202 whitespace before ']'

click here for bot help

@ewarnke

This comment has been minimized.

Copy link
Author

ewarnke commented Feb 20, 2019

I get there are behavior changes but the current default behavior is unsafe. I'm open to suggestions on how to better do this without a complete rewrite of the core modules.

@jborean93

This comment has been minimized.

Copy link
Contributor

jborean93 commented Feb 25, 2019

CI test failure - https://app.shippable.com/github/ansible/ansible/runs/109917/63/tests

{
    "changed": false,
    "invocation": {
        "module_args": {
            "_diff_peek": null,
            "_original_basename": null,
            "access_time": null,
            "access_time_format": "%Y%m%d%H%M.%S",
            "attributes": null,
            "backup": null,
            "content": null,
            "delimiter": null,
            "directory_mode": null,
            "follow": true,
            "force": false,
            "group": null,
            "mode": null,
            "modification_time": null,
            "modification_time_format": "%Y%m%d%H%M.%S",
            "owner": null,
            "path": "/root/ansible_testing/foobar",
            "recurse": false,
            "regexp": null,
            "remote_src": null,
            "selevel": null,
            "serole": null,
            "setype": null,
            "seuser": null,
            "src": null,
            "state": "absent",
            "unsafe_writes": null
        }
    },
    "msg": "rmtree failed: [Errno 39] Directory not empty: b'/root/ansible_testing/foobar'"
}

This is occurring for all the nodes running the file integration tests.

Reported remarks have been fixed

@dagwieers

This comment has been minimized.

Copy link
Member

dagwieers commented Feb 25, 2019

@ewarnke I agree, but the warning was intended for others reviewing this PR, not for you :-)

@dagwieers
Copy link
Member

dagwieers left a comment

LGTM, however this PR changes the behaviour for existing users, on the other hand it fixes a security-related issue that could cause unintended data-loss. So IMO the impact of the latter trumps the former.

We may want to issue a warning and a deprecation period, rather than an immediate break in behaviour ?

@brontide

This comment has been minimized.

Copy link

brontide commented Feb 25, 2019

Yeah, I've been thinking about abandoning these changes and working on a new pull. What's the thinking on new states, new argument, or new module?

  • States: absent_hard and absent_simple to introduce the new behaviors where absent_hard is the current behavior. That way documentation can be updated over time with the new behavior phased in. I don't like this because it makes a core module feel different.
  • Arguments: recursive_remove could be introduced to control the new behavior where it's yes by default for now and could be changed to no in a later release once people are given time to adjust. We could do a os.rmdir and if it fails, throw a warning about the changed behavior.
  • Module: Part of the problem is creation and destruction of files, links, and directories are subtly different. Maybe we should introduce a link and directory module that can better target these types of functions. That said the downside here is that you could not longer just pass a list of items to file state=absent and have them removed if they are files and directories.

I'm not tied to any one solution and the more I worked on this patch I'm just not happy with the way it's turning out. I'm fairly positive it would introduce major breakage as-is. I'm leaning towards a new argument right now since it seems like a good balance between fixing the bug and breaking existing behaviors while giving a path for people to select how directories are removed.

@brontide

This comment has been minimized.

Copy link

brontide commented Feb 25, 2019

@jborean93 Yeah, I've seen that and even tried chasing a few of the test down to add the flag. The problem is that the current behavior is unsafe and trying to find a good solution that merges cleanly and fixes the unsafe behavior without breaking piles of existing tasks is non-trivial. The fix is easy but people generally don't like breaking changes. - FYI, this is @ewarnke I just forgot to login under my work account.

@samdoran samdoran added the P3 label Feb 26, 2019

@bcoca bcoca removed the affects_2.8 label Feb 26, 2019

@ansibot

This comment has been minimized.

Copy link
Contributor

ansibot commented Feb 28, 2019

@ewarnke This PR was evaluated as a potentially problematic PR for the following reasons:

  • More than 50 changed files.

Such PR can only be merged by human. Contact a Core team member to review this PR on IRC: #ansible-devel on irc.freenode.net

click here for bot help

@felixfontein

This comment has been minimized.

Copy link
Contributor

felixfontein commented Feb 28, 2019

I disagree that this should be labelled as a bugfix. It is a behavior change (the old behavior was properly documented in the module!), and since it happens for a module with stable interface, it should be listed way more prominently in the changelog. (A porting guide entry is definitely also missing, but that was already mentioned somewhere above.)

@ewarnke ewarnke force-pushed the ewarnke:patch-2 branch from 75294cb to a8a53d9 Feb 28, 2019

@ewarnke

This comment has been minimized.

Copy link
Author

ewarnke commented Feb 28, 2019

The remaining shippable error is unrelated to the code changes.

@@ -208,6 +208,10 @@ PowerShell module options and option choices are currently case insensitive to w
specification. This behaviour is deprecated and a warning displayed to the user if a case insensitive match was found.
A future release of Ansible will make these checks case sensitive.

The file module defaults to ``recurse=no`` but didn't respect that value when deleting directories, it would always
complete a tree delete. Tasks removing directories will need to explicitly add ``recurse=yes`` if they want to remove
a directory tree. When unspecified or no, ``state=absent`` will only remove an empty directory.

This comment has been minimized.

@felixfontein

felixfontein Feb 28, 2019

Contributor
Suggested change
a directory tree. When unspecified or no, ``state=absent`` will only remove an empty directory.
a directory tree. When unspecified or set to ``no``, ``state=absent`` will only remove an empty directory.

This comment has been minimized.

@brontide

brontide Feb 28, 2019

Looks good, I'll merge it in when I'm back at my desk.

This comment has been minimized.

@ewarnke

ewarnke Mar 1, 2019

Author

Let me know if you like this wording better.

ewarnke and others added some commits Feb 20, 2019

Update file.py
Make state=absent respect recuse=False for previous state=directory.
Update file.py
Can't unlink directories, need to os.rmdir
Update initialize.yml
Fix integration so that recurse directory delete requires recurse: true
Update directory_as_dest.yml
Fix integration test
Update file.py
Update parameter checking, docs, for state=absent recuse=true.
Update file.py
Make state=absent respect recuse=False for previous state=directory.
Update file.py
Can't unlink directories, need to os.rmdir
Update lib/ansible/modules/files/file.py
Doc suggestion

Co-Authored-By: ewarnke <32573149+ewarnke@users.noreply.github.com>
Update lib/ansible/modules/files/file.py
Doc language #2

Co-Authored-By: ewarnke <32573149+ewarnke@users.noreply.github.com>

@ewarnke ewarnke force-pushed the ewarnke:patch-2 branch from 22df96e to d0f3ca7 Mar 1, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.