Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

2015.8.0: Error writing to /var/log/salt/minion? #27063

Closed
lorengordon opened this issue Sep 11, 2015 · 35 comments
Closed

2015.8.0: Error writing to /var/log/salt/minion? #27063

lorengordon opened this issue Sep 11, 2015 · 35 comments
Assignees
Labels
Bug broken, incorrect, or confusing behavior P2 Priority 2 Packaging Related to packaging of Salt, not Salt's support for package management. Platform Relates to OS, containers, platform-based utilities like FS, system based apps severity-medium 3rd level, incorrect or bad functionality, confusing and lacks a work around
Milestone

Comments

@lorengordon
Copy link
Contributor

I just installed salt v2015.8.0 from the rpm hosted at https://repo.saltstack.com/yum, and it seems it's not creating /var/log/salt before attempting to write to the minion log. This results in the warning message, "[WARNING ] Failed to open log file, do you have permission to write to /var/log/salt/minion?". I uninstalled and reinstalled, and got the same behavior. If I manually create the directory, the message goes away and the minion log is created.

I removed v2015.8.0 and installed v2015.5.3 from epel, and it seems to create the directory the first time it attempts to write to the minion log.

# salt-call --local grains.get os
[WARNING ] Failed to open log file, do you have permission to write to /var/log/salt/minion?
local:
    CentOS
# salt-call --local --versions
Salt Version:
           Salt: 2015.8.0

Dependency Versions:
         Jinja2: 2.2.1
       M2Crypto: Not Installed
           Mako: Not Installed
         PyYAML: 3.10
          PyZMQ: 14.5.0
         Python: 2.6.6 (r266:84292, Jul 23 2015, 15:22:56)
           RAET: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
   python-gnupg: Not Installed
          smmap: Not Installed
        timelib: Not Installed

System Versions:
           dist: centos 6.7 Final
        machine: x86_64
        release: 2.6.32-573.1.1.el6.x86_64
         system: CentOS 6.7 Final
@ssgward
Copy link

ssgward commented Sep 11, 2015

@lorengordon - if you do a sudo salt-call do you get the same results?

@lorengordon
Copy link
Contributor Author

Yes. I first saw the issue executing the command as a non-root user with sudo. I then switched to root and ran the command again, but the behavior was the same.

@jfindlay jfindlay added Bug broken, incorrect, or confusing behavior severity-medium 3rd level, incorrect or bad functionality, confusing and lacks a work around cannot-reproduce cannot be replicated with info/context provided P4 Priority 4 Core relates to code central or existential to Salt labels Sep 11, 2015
@jfindlay jfindlay added this to the Blocked milestone Sep 11, 2015
@jfindlay
Copy link
Contributor

@lorengordon, thanks for the report. I have tried to reproduce this on a linode CentOS 6 VM:

[root@li1046-179 ~]# salt-call --local grains.get os
local:
    CentOS
[root@li1046-179 ~]# salt --versions
Salt Version:
           Salt: 2015.8.0

Dependency Versions:
         Jinja2: unknown
       M2Crypto: Not Installed
           Mako: Not Installed
         PyYAML: 3.11
          PyZMQ: 14.5.0
         Python: 2.6.6 (r266:84292, Jul 23 2015, 15:22:56)
           RAET: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5
           cffi: Not Installed
       cherrypy: 3.2.2
       dateutil: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
   python-gnupg: Not Installed
          smmap: Not Installed
        timelib: Not Installed

System Versions:
           dist: centos 6.7 Final
        machine: x86_64
        release: 4.1.5-x86_64-linode61
         system: CentOS 6.7 Final

@jfindlay
Copy link
Contributor

@lorengordon, is it possible for you to try with a new, clean VM? Thanks.

@lorengordon
Copy link
Contributor Author

This was a clean VM, but yes, I'll try a few AMIs in AWS to see what happens and try to figure out a pattern.

@ssgward
Copy link

ssgward commented Sep 11, 2015

@lorengordon - another thing to check for us: what are your permissions on both /var/log and /var/log/salt

Second question is SELinux installed?

@lorengordon
Copy link
Contributor Author

Permissions were fine, I was otherwise able to create the directory with no problem. And I executed the install as root and ran salt-call as root. This was a fresh install and SELinux had not yet been configured in any way. And as I mentioned 2015.5.3 had no problem on the same system. But it was a custom AMI we create ourselves and we've run into oddball problems with custom AMIs before, so I will gladly try a few others that are more vanilla and report back.

@lorengordon
Copy link
Contributor Author

Oh, rereading the question, /var/log/salt did not exist until I created it manually. That was the problem that generated the warning message. Now, why didn't the directory exist or why didn't it get created on first access? That I don't know.

@lorengordon
Copy link
Contributor Author

Just tested with a generic Amazon Linux AMI (ami-0d4cfd66) and got the same behavior I reported.

@lorengordon
Copy link
Contributor Author

I can duplicate this by spinning up a vanilla instance like ami-0d4cfd66, and executing the below commands, which do nothing special, just setup the yum saltrepo and install salt.

# cat > /etc/yum.repos.d/saltstack.repo << EOF
[saltstack-repo]
name=SaltStack repo for RHEL/CentOS 6
baseurl=https://repo.saltstack.com/yum/rhel6
enabled=1
gpgcheck=1
gpgkey=https://repo.saltstack.com/yum/rhel6/SALTSTACK-GPG-KEY.pub
EOF
# yum install salt-minion           ### <<<------- Install salt ###
# ...output snipped...
Installed:
  salt-minion.noarch 0:2015.8.0-2.el6

Dependency Installed:
  PyYAML.x86_64 0:3.11-1.el6                   python-crypto.x86_64 0:2.6.1-2.el6                                 python-futures.noarch 0:3.0.3-1.el6          python-msgpack.x86_64 0:0.4.6-1.el6
  python-tornado.x86_64 0:4.2.1-1.el6          python-zmq.x86_64 0:14.5.0-2.el6                                   python26.x86_64 0:2.6.9-1.80.amzn1           python26-babel.noarch 0:0.9.4-5.1.8.amzn1
  python26-backports.x86_64 0:1.0-3.14.amzn1   python26-backports-ssl_match_hostname.noarch 0:3.4.0.2-1.12.amzn1  python26-chardet.noarch 0:2.0.1-7.7.amzn1    python26-jinja2.noarch 0:2.7.2-2.15.amzn1
  python26-libs.x86_64 0:2.6.9-1.80.amzn1      python26-markupsafe.x86_64 0:0.11-4.6.amzn1                        python26-ordereddict.noarch 0:1.1-2.5.amzn1  python26-pycurl.x86_64 0:7.19.0-17.12.amzn1
  python26-requests.noarch 0:1.2.3-5.10.amzn1  python26-setuptools.noarch 0:12.2-1.30.amzn1                       python26-six.noarch 0:1.8.0-1.23.amzn1       python26-urllib3.noarch 0:1.8.2-1.5.amzn1
  salt.noarch 0:2015.8.0-2.el6                 zeromq.x86_64 0:4.0.5-1.el6

Complete!
# salt-call --local grains.get os   ### <<<------- Demonstrate the problem, with default config ###
[WARNING ] Failed to open log file, do you have permission to write to /var/log/salt/minion?
local:
    Amazon
# salt-call --versions              ### <<<------- Print salt version ###
Salt Version:
           Salt: 2015.8.0

Dependency Versions:
         Jinja2: 2.7.2
       M2Crypto: Not Installed
           Mako: Not Installed
         PyYAML: 3.11
          PyZMQ: 14.5.0
         Python: 2.6.9 (unknown, Apr  1 2015, 18:16:00)
           RAET: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
   python-gnupg: Not Installed
          smmap: Not Installed
        timelib: Not Installed

System Versions:
           dist:
        machine: x86_64
        release: 3.14.48-33.39.amzn1.x86_64
# mkdir /var/log/salt                ### <<<------- Fix the problem ###
# salt-call --local grains.get os    ### <<<------- Demonstrate the problem is fixed ###
local:
    Amazon

@s0undt3ch
Copy link
Collaborator

The issue here is that the verify_env config setting which used to be True was switched to False for CLI initialization performance reasons. The drawback here is that salt does not try to create missing directories or check for proper permissions on required directories.

The salt packages, in this case, should make sure those directories are created at installation time.

So, as it is now, this is a packaging issue.

@lorengordon
Copy link
Contributor Author

@s0undt3ch How can I test that? I tried just setting verify_env to True in my minion config, but it didn't help.

# rm -rf /var/log/salt
# grep verify_env /etc/salt/minion
verify_env: True
# salt-call --local grains.get os
[WARNING ] Failed to open log file, do you have permission to write to /var/log/salt/minion?
local:
    Amazon

@s0undt3ch
Copy link
Collaborator

Hmm... that should have worked.... I wonder if we're trying to write to the log file before verify env is executed....

@jfindlay jfindlay added Packaging Related to packaging of Salt, not Salt's support for package management. P2 Priority 2 and removed P4 Priority 4 labels Sep 14, 2015
@jfindlay jfindlay modified the milestones: Approved, Blocked Sep 14, 2015
@cachedout
Copy link
Contributor

I forgot that we pulled out verify_env because it was slowing down salt-call. I now agree this is something that needs to be handled on the packaging layer.

@cachedout
Copy link
Contributor

Re-assinging to @dmurphy18 per his request.

@cachedout cachedout removed the Core relates to code central or existential to Salt label Sep 18, 2015
@DmitryKuzmenko
Copy link
Contributor

Fixed the io operation error in #31544

@cachedout cachedout added fixed-pls-verify fix is linked, bug author to confirm fix and removed cannot-reproduce cannot be replicated with info/context provided labels Feb 29, 2016
@meggiebot meggiebot modified the milestones: B 1, B 2 Feb 29, 2016
@meggiebot meggiebot removed the fixed-pls-verify fix is linked, bug author to confirm fix label Mar 7, 2016
@meggiebot meggiebot modified the milestones: B 0, B 1 Mar 14, 2016
@dmurphy18 dmurphy18 modified the milestones: B -1, B 0 Mar 28, 2016
@meggiebot meggiebot modified the milestones: B -2, B -1 Apr 12, 2016
@dmurphy18 dmurphy18 modified the milestones: B -3, B -2 Apr 25, 2016
@dmurphy18 dmurphy18 modified the milestones: C 10, B -3 May 9, 2016
@dmurphy18 dmurphy18 modified the milestones: C 7, C 10 Jun 6, 2016
@dmurphy18 dmurphy18 modified the milestones: C 6, C 7 Jul 18, 2016
@dmurphy18 dmurphy18 modified the milestones: C 5, C 6 Aug 8, 2016
@dmurphy18
Copy link
Contributor

@lorengordon @ghost @jmo-learnosity I want to close this issue since testing with the latest release 2016.3.2 the issues with /var/log/salt/minion are resolved. Let me know if there are any objections to closing this issue.

@lorengordon
Copy link
Contributor Author

@dmurphy18, ok, I'll test 2016.3.2 with this in mind shortly.

@lorengordon
Copy link
Contributor Author

Yep, looks good now. /var/log/salt is created, and the error message is no longer reported. Closing, thanks!

# salt-call --local grains.get os
local:
    CentOS
# salt-call --versions-report
Salt Version:
           Salt: 2016.3.2

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.3
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.6.6 (r266:84292, Jul 23 2015, 15:22:56)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: 14.5.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5

System Versions:
           dist: centos 6.8 Final
        machine: x86_64
        release: 2.6.32-642.3.1.el6.x86_64
         system: Linux
        version: CentOS 6.8 Final

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug broken, incorrect, or confusing behavior P2 Priority 2 Packaging Related to packaging of Salt, not Salt's support for package management. Platform Relates to OS, containers, platform-based utilities like FS, system based apps severity-medium 3rd level, incorrect or bad functionality, confusing and lacks a work around
Projects
None yet
Development

No branches or pull requests

9 participants