New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stat module very slow checking status of a large file #24858

Closed
edrozenberg opened this Issue May 20, 2017 · 3 comments

Comments

Projects
None yet
3 participants
@edrozenberg

edrozenberg commented May 20, 2017

ISSUE TYPE
  • Bug Report
COMPONENT NAME

stat module

ANSIBLE VERSION
ansible 2.3.0.0
  config file = 
  configured module search path = Default w/o overrides
  python version = 2.7.13 (default, Dec 18 2016, 07:03:39) [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)]
CONFIGURATION
OS / ENVIRONMENT

MacOS 10.12.5 (16F73) controlling Linux Slackware64 14.2 server

SUMMARY

The 'stat' module runs very slowly on a large file. Takes more than 45 seconds to return stat on a 10GB swapfile.

STEPS TO REPRODUCE
# Check for presence of swap file
- name: swap file check status
  stat:
    path: '{{ swap_file }}'
  register: swapfilecheck

The swapfile being checked:

# ls -al /var/swap/swap0 
-rw------- 1 root root 10G May 20 02:35 /var/swap/swap0
EXPECTED RESULTS

stat module takes around a second to run

ACTUAL RESULTS

stat module takes around 45 seconds to run on a 10GB file.
stat module takes around 6 seconds to run on a 1GB file.

Time for stat to run appears proportional to size of file.

ok: [myserver] => {
    "changed": false, 
    "invocation": {
        "module_args": {
            "checksum_algorithm": "sha1", 
            "follow": false, 
            "get_attributes": true, 
            "get_checksum": true, 
            "get_md5": true, 
            "get_mime": true, 
            "path": "/var/swap/swap0"
        }
    }, 
    "stat": {
        "atime": 1495227955.2459276, 
        "attr_flags": "", 
        "attributes": [], 
        "block_size": 4096, 
        "blocks": 20971528, 
        "charset": "binary", 
        "checksum": "d753959ab4627d37b5d1cb45cf6d9f8383ce9911", 
        "ctime": 1495227952.8889482, 
        "dev": 2307, 
        "device_type": 0, 
        "executable": false, 
        "exists": true, 
        "gid": 0, 
        "gr_name": "root", 
        "inode": 12451842, 
        "isblk": false, 
        "ischr": false, 
        "isdir": false, 
        "isfifo": false, 
        "isgid": false, 
        "islnk": false, 
        "isreg": true, 
        "issock": false, 
        "isuid": false, 
        "md5": "c1352ac7d244e2d96e0c883448261cfc", 
        "mimetype": "application/octet-stream", 
        "mode": "0600", 
        "mtime": 1495227952.8889482, 
        "nlink": 1, 
        "path": "/var/swap/swap0", 
        "pw_name": "root", 
        "readable": true, 
        "rgrp": false, 
        "roth": false, 
        "rusr": true, 
        "size": 10737418240, 
        "uid": 0, 
        "version": "340312571", 
        "wgrp": false, 
        "woth": false, 
        "writeable": true, 
        "wusr": true, 
        "xgrp": false, 
        "xoth": false, 
        "xusr": false
    }
}
@ansibot

This comment has been minimized.

Show comment
Hide comment
@ansibot
Contributor

ansibot commented May 20, 2017

@edrozenberg

This comment has been minimized.

Show comment
Hide comment
@edrozenberg

edrozenberg May 20, 2017

My guess is issue is due to the checksumming always being done on a file. If this is the problem, a possible solution is to add a way to disable checksumming such as by adding a 'none' option to checksumming:

checksum_algorithm = none

Setting checksumming off would remove the "checksum" item from the stat available return vars.

edrozenberg commented May 20, 2017

My guess is issue is due to the checksumming always being done on a file. If this is the problem, a possible solution is to add a way to disable checksumming such as by adding a 'none' option to checksumming:

checksum_algorithm = none

Setting checksumming off would remove the "checksum" item from the stat available return vars.

@edrozenberg

This comment has been minimized.

Show comment
Hide comment
@edrozenberg

edrozenberg May 20, 2017

Just read the stat doc page in more detail.

Setting:

    get_checksum: False
    get_md5: False

Fixes the issue with slowdown.

edrozenberg commented May 20, 2017

Just read the stat doc page in more detail.

Setting:

    get_checksum: False
    get_md5: False

Fixes the issue with slowdown.

@bcoca bcoca removed the needs_triage label May 22, 2017

@ansibot ansibot added bug and removed bug_report labels Mar 7, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment