Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Troubleshoot file not found and permission denied errors during snapshot creation
EBS snapshots are a global entity. Within Eucalyptus this statement translates to snapshots being available across multiple Eucalyptus clusters regardless of the cluster they originate from. In order to make this happen, the Storage Controller (SC) uploads the snapshot to Object Storage Gateway (S3 equivalent) during snapshot creation. They are downloaded by Storage Controllers in other Eucalyptus clusters on demand (i.e. when a volume is requested from the snapshot in the cluster).
When EBS storage is backed by SANs such as Netapp, EMC or Equallogic, SC orchestrates the creation of snapshot and exports it as a block device to the SC host.It is necessary that SC be able to read from the block device in order to upload the contents to OSG. Eucalyptus provides for udev rules (55-openiscsi.rules, 12-dm-permissions.rules) and udev script (iscsidev.sh) that assign the ownership of the block device to the 'eucalyptus' user. If the udev rules/script are missing or the udev script does not account for the storage SAN IQN, the ownership of the block device is not assigned to Eucalyptus and may cause snapshot creation to fail with 'File not found (permission denied)' error.
Follow the instructions for installing the udev rules and udev script
The following debug steps are to be executed manually without the use of Eucalyptus . This helps isolate the issue and pinpoint it to a specific component. These steps can be also be used to troubleshoot the issue and fix the issues as you go.
- Log out of all iSCSI sessions on the SC host logging the error
iscsiadm --mode session --logout
Delete all the iscsiadm nodeson the system
iscsiadm --mode node --op=delete
- Verify that 'eucalyptus' user exists on that system from the output of
If eucalyptus is not listed as one of the users, add it
Verify that the necessary udev rules and the udev script are in place. Refer to the .
Create a volume on the SAN backend, DO NOT use Eucalyptus to create the volume
Establish an iSCSI connection on the SC host to the SAN volume using static targets method
After successfully connecting to the SAN volume, check the block device (/dev/sd) of the attachment from the output of
iscsiadm -m session -P3
- List the permissions of the block device /dev/sdX and verify that ownership is assigned to 'eucalyptus' user.
If the ownership is not assigned to 'eucalyptus' user, it is safe to assume that the issue might be related to udev rules or udev script. Further debugging of udev may be necessary.
Follow the steps below to debug udev artifacts responsible for adjusting the ownership of block device attachments
- Verify that all the necessary udev artifacts discussed in the are in place
- Bump up the udev log level to 'DEBUG'. udev generally logs to syslog at /var/log/messages
udevadm control --log-priority=debug
- Reload the udev rules by
udevadm control --reload-rules
- Repeat 1 through 6 steps in the above section and get the /dev/sd block device of the attachment
- Look for udev 55-openiscsi.rules invocation in /var/log/messages.If you find the script invocation, proceed to next step
If you don't see the script being invoked,verify that 55-openiscsi.rules matches the /dev/sd block device correctly. This can be checked by triggering udev rules in dry-run mode
udevadm trigger --verbose --dry-run --type=devices --subsystem-match=block
/dev/sd block device obtained from step 4 should be listed as one of the matches in the output of the above command. If not, the udev rule may not be accurate. Refer to 55-openiscsi.rules in Eucalyptus github repo
- Check for the return code from the udev rule invocation. If the script does not return success, i.e. 0, there might be an issue with the iscsidev.sh script. Copy the invocation from the log and try executing the iscisdev.sh script by itself to debug it further. Refer to iscsidev.sh script in Eucalyptus github repo