nfs::mount ensure: mounted does not work #45

ipoppo · 2015-03-03T12:56:43Z

Specify ensure: mounted will cause the following log

Notice: /Stage[main]/Nfs/Types::Mount[package-mount]/Mount[/mnt/package]/ensure: ensure changed 'unmounted' to 'mounted'
Error: /Stage[main]/Nfs/Types::Mount[package-mount]/Mount[/mnt/package]: Could not evaluate: Execution of '/bin/mount /mnt/package' returned 32: mount.nfs: an incorrect mount option was specified

Specify ensure: present does not generate log and create /mnt/package but device not mounted (as expected)

# sample common.yaml
nfs::mounts:
  package-mount:
    name:    "%{hiera('package::mount')}"
    device:  "%{hiera('package::hostname')}:%{hiera('package::export')}"
    ensure:  mounted
    options: rw,sync,no_root_squash
    fstype:  nfs

The text was updated successfully, but these errors were encountered:

ghoneycutt · 2015-03-03T18:04:59Z

Looks like your NFS implementation does not like the commands given. Try running with -d for debug and see the exact command being run. Then try that command by itself and note the return code. Would also help to know what version of this module are you using and what platform you are running this on.

ipoppo · 2015-03-05T03:06:30Z

This installation is hosted on Oracle Linux Server release 6.3, kernel 2.6.32-279.11.1.el6.x86_64 both master and agent pair

# puppet --version
3.4.2

# puppet module list
/etc/puppet/modules
├── ghoneycutt-common (v1.2.0)
├── ghoneycutt-nfs (v1.8.0)
├── ghoneycutt-rpcbind (v1.4.0)
├── ghoneycutt-types (v1.6.0)

The result of debug is shown below

Debug: Failed to load library 'msgpack' for feature 'msgpack'
Debug: file_metadata supports formats: pson b64_zlib_yaml yaml raw
Debug: Finishing transaction 23936455727840
Info: Loading facts in /etc/puppet/modules/concat/lib/facter/concat_basedir.rb
Info: Loading facts in /etc/puppet/modules/oradb/lib/facter/oracle_database_homes.rb
Debug: oradb opatch is
Info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/puppet_vardir.rb
Info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/root_home.rb
Info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/pe_version.rb
Info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/facter_dot_d.rb
Info: Loading facts in /etc/puppet/modules/oracle/lib/facter/asm_facts.rb
Info: Loading facts in /var/lib/puppet/lib/facter/root_home.rb
Info: Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rb
Info: Loading facts in /var/lib/puppet/lib/facter/pe_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rb
Info: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb
Debug: Failed to load library 'msgpack' for feature 'msgpack'
Debug: catalog supports formats: pson dot b64_zlib_yaml yaml raw
Debug: Executing '/bin/rpm --version'
Debug: Executing '/bin/rpm -ql rpm'
Debug: Executing '/bin/rpm --version'
Info: Caching catalog for puppet-test-appserver
Debug: Puppet::Type::Package::ProviderUrpmi: file urpmi does not exist
Debug: Puppet::Type::Package::ProviderSunfreeware: file pkg-get does not exist
Debug: Puppet::Type::Package::ProviderSun: file /usr/sbin/pkgadd does not exist
Debug: Puppet::Type::Package::ProviderPortupgrade: file /usr/local/sbin/portupgrade does not exist
Debug: Puppet::Type::Package::ProviderZypper: file /usr/bin/zypper does not exist
Debug: Puppet::Type::Package::ProviderRug: file /usr/bin/rug does not exist
Debug: Puppet::Type::Package::ProviderAptitude: file /usr/bin/aptitude does not exist
Debug: Puppet::Type::Package::ProviderDpkg: file /usr/bin/dpkg does not exist
Debug: Puppet::Type::Package::ProviderPkg: file /usr/bin/pkg does not exist
Debug: Puppet::Type::Package::ProviderApt: file /usr/bin/apt-get does not exist
Debug: Puppet::Type::Package::ProviderOpenbsd: file pkg_add does not exist
Debug: Puppet::Type::Package::ProviderAix: file /usr/bin/lslpp does not exist
Debug: Puppet::Type::Package::ProviderPortage: file /usr/bin/eix-update does not exist
Debug: Puppet::Type::Package::ProviderAptrpm: file apt-get does not exist
Debug: Puppet::Type::Package::ProviderFink: file /sw/bin/fink does not exist
Debug: Puppet::Type::Package::ProviderFreebsd: file /usr/sbin/pkg_add does not exist
Debug: Puppet::Type::Package::ProviderPacman: file /usr/bin/pacman does not exist
Debug: Puppet::Type::Package::ProviderHpux: file /usr/sbin/swinstall does not exist
Debug: Puppet::Type::Package::ProviderNim: file /usr/sbin/nimclient does not exist
Debug: Puppet::Type::Package::ProviderPkgin: file pkgin does not exist
Debug: Puppet::Type::Package::ProviderPorts: file /usr/local/sbin/portupgrade does not exist
Debug: Puppet::Type::Package::ProviderUp2date: file /usr/sbin/up2date-nox does not exist
Debug: Puppet::Type::Package::ProviderOpkg: file opkg does not exist
Debug: Puppet::Type::Service::ProviderRunit: file /usr/bin/sv does not exist
Debug: Puppet::Type::Service::ProviderLaunchd: file /bin/launchctl does not exist
Debug: Puppet::Type::Service::ProviderSystemd: file systemctl does not exist
Debug: Puppet::Type::Service::ProviderDaemontools: file /usr/bin/svc does not exist
Debug: Puppet::Type::Service::ProviderGentoo: file /sbin/rc-update does not exist
Debug: Puppet::Type::Service::ProviderOpenrc: file /bin/rc-status does not exist
Debug: Puppet::Type::Service::ProviderDebian: file /usr/sbin/update-rc.d does not exist
Debug: Creating default schedules
Debug: Loaded state in 0.02 seconds
Debug: /Stage[main]/Nfs::Idmap/Service[idmapd_service]/subscribe: subscribes to File[idmapd_conf]
Debug: /Stage[main]/Nfs::Idmap/File[idmapd_conf]/require: requires Package[nfs-utils-lib]
Debug: /Stage[main]/Rpcbind/Service[rpcbind_service]/require: requires Package[rpcbind]
Debug: /Stage[main]/Nfs/Types::Mount[package-mount]/Mount[/mnt/package]/require: requires Common::Mkdir_p[/mnt/package]
Debug: /Stage[main]/Nfs/Service[nfs_service]/subscribe: subscribes to Package[nfs-utils]
Debug: /Stage[main]/Nfs/before: requires Class[Pmdm::Deploy]
Info: Applying configuration version '1425524112'
Debug: Prefetching yum resources for package
Debug: Executing '/bin/rpm --version'
Debug: Executing '/bin/rpm -qa --nosignature --nodigest --qf '%{NAME} %|EPOCH?{%{EPOCH}}:{0}| %{VERSION} %{RELEASE} %{ARCH} :DESC: %{SUMMARY}\n''
Debug: Executing '/sbin/service rpcidmapd status'
Debug: Executing '/sbin/chkconfig rpcidmapd'
Debug: Exec[mkdir_p-/mnt/package](provider=posix): Executing check 'test -d /mnt/package'
Debug: Executing 'test -d /mnt/package'
Debug: Executing '/sbin/service rpcbind status'
Debug: Executing '/sbin/chkconfig rpcbind'
Debug: Executing '/sbin/service nfs status'
Debug: Executing '/sbin/chkconfig nfs'
Debug: Prefetching parsed resources for mount
Debug: Executing '/bin/mount'
Notice: /Stage[main]/Nfs/Types::Mount[package-mount]/Mount[/mnt/package]/ensure: ensure changed 'unmounted' to 'mounted'
Debug: Executing '/bin/mount /mnt/package'
Error: /Stage[main]/Nfs/Types::Mount[package-mount]/Mount[/mnt/package]: Could not evaluate: Execution of '/bin/mount /mnt/package' returned 32: mount.nfs: an incorrect mount option was specified

execute /bin/mount /mnt/package yields same result

# /bin/mount /mnt/package
mount.nfs: an incorrect mount option was specified
# echo $?
32

ipoppo · 2015-03-12T03:24:35Z

Hi, is there any finding on the issue updated?

ghoneycutt · 2015-03-12T03:45:40Z

It seems you have an incorrect mount option.. though the entry in Hiera seems fine to me. I'm unable to duplicate the issue.

ghoneycutt · 2015-03-12T03:49:07Z

Try not specifying any options and see if it mounts, then add an option, re-run puppet and repeat.

ipoppo · 2015-03-13T10:05:11Z

/bin/mount /mnt/package does not happy with no option command

helperton · 2015-11-04T19:13:44Z

We have a similar issue with OEL 5 and 6, mount and umount work great, but when Puppet goes to 'mount -o remount /mount'. This seemed to happen when we moved from Netapp to Isilon backend.

Error: /Stage[main]/Puppet_nfs::Main/Mount[/mount]: Failed to call refresh: Execution of '/bin/mount -o remount /mount' returned 32: mount.nfs: an incorrect mount option was specified

Error: /Stage[main]/Puppet_nfs::Main/Mount[/mount]: Execution of '/bin/mount -o remount /mount' returned 32: mount.nfs: an incorrect mount option was specified

[root@tstpupt902 ~]# mount -o remount /mount
mount.nfs: an incorrect mount option was specified

fstab entry (about as basic as you can get):
nfsserver:/ifs/share /mount nfs defaults 0 0

ghoneycutt · 2015-11-04T19:36:08Z

Hmm.. it appears that the fstab entry is incorrect, though I'm stumped. Any ideas?

helperton · 2015-11-04T19:43:16Z

If it were incorrect, then it wouldn't even mount (which it does), it's 'remount' which complains. We're still trying to figure out what's up.

helperton · 2015-11-04T20:41:46Z

So, we determined that for our issue, it indeed was the Isilon backend causing the issue. What happens is that a initial mount goes to one Isilon load balanced head, but when a subsequest 'mount -o remount' is asked for, that request is balanced to a different head which has no idea that a mount already existed and rejects the remount request.

ghoneycutt · 2015-11-04T20:50:18Z

@ipoppo what type of nfs server are you using?

ghoneycutt · 2015-11-04T20:55:05Z

@helperton is this something you fix entirely on the nfs server or are there special options that are needed for the nfs client? Either way, I'd be happy to document all of this info along with a fix in the README to help others.

helperton · 2015-11-04T21:16:09Z

We have no fix yet. Here's my troubleshooting steps though:

Since Isilon load balances nfs requests which come to it, the initial mount goes to one head, but the ‘remount’ command hits a different host, which doesn’t seem to know about the original mount which existed on the head which originally serviced the request.

If I go into my /etc/hosts and hard code in an IP for a specific head (e.g. 10.0.0.61 somenas.yourcomp.org somenas) so it only ever talks to this head, then a ‘remount’ works just fine.

Here’s how I found how it was going elsewhere on a remount:

I ran an strace on both a mount and mount with -o remount added and saw this:

Here’s the mount:

mount("somenas.somecomp.org:/ifs/mount", "/mount", "nfs", MS_RDONLY|MS_NOSUID, "soft,vers=4,addr=10.0.0.53,clientaddr=10.1.0.71") = 0

Here’s the mount with -o remount added:

mount("somenas.somecomp.org:/ifs/mount", "/mount", 0x7f8abd3651bb, MS_RDONLY|MS_NOSUID|MS_REMOUNT, "soft,addr=10.0.0.61") = -1 EINVAL (Invalid argument)

You can see it going to a different ‘addr’. In the first request it goes to 10.0.0.53 and in the 2nd request it goes to 10.0.0.61. I guess the Isilon heads don't share state info, or at least don't share it quick enough for the other head to know about this mount before the remount happens (which it does almost immediately with the Puppet mount provider).

helperton · 2015-11-04T21:23:13Z

A possible fix on the Puppet side (inside the mount provider) would be for it to first scan the 'addr' of the host the mount is currently connected to. Then when it issues the 'remount', to provide, 'addr=(the same ip it's already connected to)'. This way, it doesn't allow the mount command to do a fresh dns lookup.

ghoneycutt · 2015-11-04T21:42:35Z

@helperton The mount provider[1] does not do any resolving of a potential hostname and thus cannot cache that value which I believe would lead to a system that was not idempotent. It is sloppy that Isilon returns a different name instead of hiding that and routing the request for you. Anyhow, I think the best way to tackle this is to use a fixed name with an entry in /etc/hosts. You can use https://github.com/ghoneycutt/puppet-module-hosts for this. A simple approach would be to identify nodes that share an nfs server and then use hiera to ensure the mount and the host entry.

[1] - https://github.com/puppetlabs/puppet/blob/master/lib/puppet/provider/mount.rb

ghoneycutt · 2015-11-04T21:43:16Z

Going to close this as the issue is with the NFS server implementation and not the nfs module and we have a work around.

helperton · 2015-11-04T21:47:59Z

I never said the mount provider did any resolving of hostnames. The OS command 'mount' does that. My point is that the mount provider could do 'cat /proc/mounts' and look for the current value of 'addr=' for the mount in question, prior to issuing the 'remount' and provide 'addr=' with the remount so the OS mount command doesn't need to do a fresh DNS lookup.

ghoneycutt · 2015-11-04T21:50:49Z

@helperton @ipoppo Thanks for your help on this!!

ghoneycutt · 2015-11-04T21:51:56Z

@helperton If you want to go that route, I would open a bug with Puppet Labs on the mount provider and see if they are receptive to that implementation. If you do, please add me as a watcher, as I'm curious :)

helperton · 2015-11-04T22:24:56Z

We're an Enterprise customer, so I'm not sure you can track the issue, but here it is:

https://support.puppetlabs.com/requests/14769

ghoneycutt closed this as completed Nov 4, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nfs::mount ensure: mounted does not work #45

nfs::mount ensure: mounted does not work #45

ipoppo commented Mar 3, 2015

ghoneycutt commented Mar 3, 2015

ipoppo commented Mar 5, 2015

ipoppo commented Mar 12, 2015

ghoneycutt commented Mar 12, 2015

ghoneycutt commented Mar 12, 2015

ipoppo commented Mar 13, 2015

helperton commented Nov 4, 2015

ghoneycutt commented Nov 4, 2015

helperton commented Nov 4, 2015

helperton commented Nov 4, 2015

ghoneycutt commented Nov 4, 2015

ghoneycutt commented Nov 4, 2015

helperton commented Nov 4, 2015

helperton commented Nov 4, 2015

ghoneycutt commented Nov 4, 2015

ghoneycutt commented Nov 4, 2015

helperton commented Nov 4, 2015

ghoneycutt commented Nov 4, 2015

ghoneycutt commented Nov 4, 2015

helperton commented Nov 4, 2015

nfs::mount ensure: mounted does not work #45

nfs::mount ensure: mounted does not work #45

Comments

ipoppo commented Mar 3, 2015

ghoneycutt commented Mar 3, 2015

ipoppo commented Mar 5, 2015

ipoppo commented Mar 12, 2015

ghoneycutt commented Mar 12, 2015

ghoneycutt commented Mar 12, 2015

ipoppo commented Mar 13, 2015

helperton commented Nov 4, 2015

ghoneycutt commented Nov 4, 2015

helperton commented Nov 4, 2015

helperton commented Nov 4, 2015

ghoneycutt commented Nov 4, 2015

ghoneycutt commented Nov 4, 2015

helperton commented Nov 4, 2015

helperton commented Nov 4, 2015

ghoneycutt commented Nov 4, 2015

ghoneycutt commented Nov 4, 2015

helperton commented Nov 4, 2015

ghoneycutt commented Nov 4, 2015

ghoneycutt commented Nov 4, 2015

helperton commented Nov 4, 2015