Failed to install via Cloud formation template #238

Closed
coolbox opened this Issue Apr 10, 2015 · 9 comments

Projects

None yet

6 participants

@coolbox
coolbox commented Apr 10, 2015

I've successfully setup the Cloud Formation stack on AWS. I had a fully functioning auto-scaling group setup which was scaling my DynamoDB table. For testing purposes i terminated the EC2 instance to check another was launched. Another was successfully launched but dynamic-dynamodb fails to install.

I can see in the installer.log, the line: Failed to install dynamic-dynamodb package from pip repository.

I've ssh'd onto the ec2 instance and manually tried to install via pip, and saw the following:

pip install -U -r /etc/dynamic-dynamodb/requirements.txt
Collecting dynamic-dynamodb<2.0.0,>=1.7.0 (from -r /etc/dynamic-dynamodb/requirements.txt (line 1))
  Using cached dynamic-dynamodb-1.20.4.tar.gz
Collecting boto>=2.29.1 (from dynamic-dynamodb<2.0.0,>=1.7.0->-r /etc/dynamic-dynamodb/requirements.txt (line 1))
  Using cached boto-2.38.0-py2.py3-none-any.whl
Collecting requests>=0.14.1 (from dynamic-dynamodb<2.0.0,>=1.7.0->-r /etc/dynamic-dynamodb/requirements.txt (line 1))
  Using cached requests-2.6.0-py2.py3-none-any.whl
Collecting logutils>=0.3.3 (from dynamic-dynamodb<2.0.0,>=1.7.0->-r /etc/dynamic-dynamodb/requirements.txt (line 1))
  Using cached logutils-0.3.3.tar.gz
Collecting retrying>=1.3.3 (from dynamic-dynamodb<2.0.0,>=1.7.0->-r /etc/dynamic-dynamodb/requirements.txt (line 1))
  Using cached retrying-1.3.3.tar.gz
Collecting six>=1.7.0 (from retrying>=1.3.3->dynamic-dynamodb<2.0.0,>=1.7.0->-r /etc/dynamic-dynamodb/requirements.txt (line 1))
  Using cached six-1.9.0-py2.py3-none-any.whl
Installing collected packages: boto, requests, logutils, six, retrying, dynamic-dynamodb
  Found existing installation: boto 2.27.0
    Uninstalling boto-2.27.0:
      Exception:
      Traceback (most recent call last):
        File "/usr/local/lib/python2.6/site-packages/pip-6.1.1-py2.6.egg/pip/basecommand.py", line 246, in main
          status = self.run(options, args)
        File "/usr/local/lib/python2.6/site-packages/pip-6.1.1-py2.6.egg/pip/commands/install.py", line 352, in run
          root=options.root_path,
        File "/usr/local/lib/python2.6/site-packages/pip-6.1.1-py2.6.egg/pip/req/req_set.py", line 687, in install
          requirement.uninstall(auto_confirm=True)
        File "/usr/local/lib/python2.6/site-packages/pip-6.1.1-py2.6.egg/pip/req/req_install.py", line 730, in uninstall
          paths_to_remove.remove(auto_confirm)
        File "/usr/local/lib/python2.6/site-packages/pip-6.1.1-py2.6.egg/pip/req/req_uninstall.py", line 126, in remove
          renames(path, new_path)
        File "/usr/local/lib/python2.6/site-packages/pip-6.1.1-py2.6.egg/pip/utils/__init__.py", line 292, in renames
          shutil.move(old, new)
        File "/usr/lib64/python2.6/shutil.py", line 258, in move
          rmtree(src)
        File "/usr/lib64/python2.6/shutil.py", line 217, in rmtree
          onerror(os.remove, fullname, sys.exc_info())
        File "/usr/lib64/python2.6/shutil.py", line 215, in rmtree
          os.remove(fullname)
      OSError: [Errno 13] Permission denied: '/usr/lib/python2.6/site-packages/boto/https_connection.pyc'

Do you have any suggestions on how i can resolve this please?
Is this an error others are experiencing or is this special to me for some reason?

Thanks.

@sebdah
Owner
sebdah commented Apr 10, 2015

Thanks for the report. It seems like the reinstall lacks root privileges.
I'm busy at the moment and I'll need to take a closer look into this.

Will update here when I've looked at it.

On Fri, Apr 10, 2015, 16:29 Pete Roome notifications@github.com wrote:

I've successfully setup the Cloud Formation stack on AWS. I had a fully
functioning auto-scaling group setup which was scaling my DynamoDB table.
For testing purposes i terminated the EC2 instance to check another was
launched. Another was successfully launched but dynamic-dynamodb fails to
install.

I can see in the installer.log, the line: Failed to install
dynamic-dynamodb package from pip repository.

I've ssh'd onto the ec2 instance and manually tried to install via pip,
and saw the following:

pip install -U -r /etc/dynamic-dynamodb/requirements.txt
Collecting dynamic-dynamodb<2.0.0,>=1.7.0 (from -r /etc/dynamic-dynamodb/requirements.txt (line 1))
Using cached dynamic-dynamodb-1.20.4.tar.gz
Collecting boto>=2.29.1 (from dynamic-dynamodb<2.0.0,>=1.7.0->-r /etc/dynamic-dynamodb/requirements.txt (line 1))
Using cached boto-2.38.0-py2.py3-none-any.whl
Collecting requests>=0.14.1 (from dynamic-dynamodb<2.0.0,>=1.7.0->-r /etc/dynamic-dynamodb/requirements.txt (line 1))
Using cached requests-2.6.0-py2.py3-none-any.whl
Collecting logutils>=0.3.3 (from dynamic-dynamodb<2.0.0,>=1.7.0->-r /etc/dynamic-dynamodb/requirements.txt (line 1))
Using cached logutils-0.3.3.tar.gz
Collecting retrying>=1.3.3 (from dynamic-dynamodb<2.0.0,>=1.7.0->-r /etc/dynamic-dynamodb/requirements.txt (line 1))
Using cached retrying-1.3.3.tar.gz
Collecting six>=1.7.0 (from retrying>=1.3.3->dynamic-dynamodb<2.0.0,>=1.7.0->-r /etc/dynamic-dynamodb/requirements.txt (line 1))
Using cached six-1.9.0-py2.py3-none-any.whl
Installing collected packages: boto, requests, logutils, six, retrying, dynamic-dynamodb
Found existing installation: boto 2.27.0
Uninstalling boto-2.27.0:
Exception:
Traceback (most recent call last):
File "/usr/local/lib/python2.6/site-packages/pip-6.1.1-py2.6.egg/pip/basecommand.py", line 246, in main
status = self.run(options, args)
File "/usr/local/lib/python2.6/site-packages/pip-6.1.1-py2.6.egg/pip/commands/install.py", line 352, in run
root=options.root_path,
File "/usr/local/lib/python2.6/site-packages/pip-6.1.1-py2.6.egg/pip/req/req_set.py", line 687, in install
requirement.uninstall(auto_confirm=True)
File "/usr/local/lib/python2.6/site-packages/pip-6.1.1-py2.6.egg/pip/req/req_install.py", line 730, in uninstall
paths_to_remove.remove(auto_confirm)
File "/usr/local/lib/python2.6/site-packages/pip-6.1.1-py2.6.egg/pip/req/req_uninstall.py", line 126, in remove
renames(path, new_path)
File "/usr/local/lib/python2.6/site-packages/pip-6.1.1-py2.6.egg/pip/utils/init.py", line 292, in renames
shutil.move(old, new)
File "/usr/lib64/python2.6/shutil.py", line 258, in move
rmtree(src)
File "/usr/lib64/python2.6/shutil.py", line 217, in rmtree
onerror(os.remove, fullname, sys.exc_info())
File "/usr/lib64/python2.6/shutil.py", line 215, in rmtree
os.remove(fullname)
OSError: [Errno 13] Permission denied: '/usr/lib/python2.6/site-packages/boto/https_connection.pyc'

Do you have any suggestions on how i can resolve this please?
Is this an error others are experiencing or is this special to me for some
reason?

Thanks.

Reply to this email directly or view it on GitHub
#238.

@sebdah
Owner
sebdah commented Apr 10, 2015

(Be aware - Friday afternoon hack below!)

As said, I'm busy. But I couldn't resist looking at it quickly. If you have some spare time you could try the following CF template: https://gist.github.com/sebdah/164a128f05f7fa4ed26b

The only change is that I enforce sudo when installing dynamic-dynamodb.

@coolbox
coolbox commented Apr 13, 2015

@sebdah thanks very much for taking a look. I'll give this a try now.

@coolbox
coolbox commented Apr 13, 2015

@sebdah this hasn't worked unfortunately. Still no install of on boot after a termination of the EC2 instance.

The installer.log is returning the same error. Manually running the install command, i see this:

sudo pip install -U -r /etc/dynamic-dynamodb/requirements.txt
sudo: pip: command not found

This indicates to me there is a problem with this line in the template:

"easy_install pip || error_exit 'Failed to install pip'",

…continuing to debug this and will report back with anything i find.

UPDATE:
Running which pip quickly revealed what the path to pip was. Running the following on the EC2 instance resulted in a successful install of dynamic-dynamodb:

sudo /usr/local/bin/pip install -U -r /etc/dynamic-dynamodb/requirements.txt

I've updated the Cloudformation template with the line above, but this still isn't working on reboot of the terminated EC2 instance. I still see this line in the installer.log:

Failed to install dynamic-dynamodb package from pip repository
@dhannon
dhannon commented Apr 15, 2015

I am experiencing this issue also.
the dynamic-dynamodb service does not install

Any help would be greatly appreciated

@ben-bourdin451

I'm having the same problem.

I do think this is a permissions problem but as @coolbox pointed out, adding sudo doesn't work. Surprisingly though running:

sudo /usr/local/bin/pip install -U -r /etc/dynamic-dynamodb/requirements.txt

on the server works fine, so not sure if this is to do specifically with AWS.

Here is my cloud init log if it helps:

Apr 20 13:15:21 cloud-init[1329]: util.py[DEBUG]: Running command ['/var/lib/cloud/instance/scripts/part-001'] with allowed return codes [0] (shell=True, capture=False)
Apr 20 13:15:29 cloud-init[1329]: util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-001 [1]
Apr 20 13:15:29 cloud-init[1329]: util.py[DEBUG]: Failed running /var/lib/cloud/instance/scripts/part-001 [1]
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/cloudinit/util.py", line 637, in runparts
    subp([exe_path], capture=False, shell=True)
  File "/usr/lib/python2.6/site-packages/cloudinit/util.py", line 1528, in subp
    cmd=args)
ProcessExecutionError: Unexpected error while running command.
Command: ['/var/lib/cloud/instance/scripts/part-001']
Exit code: 1
Reason: -
Stdout: ''
Stderr: ''
Apr 20 13:15:29 cloud-init[1329]: cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
Apr 20 13:15:29 cloud-init[1329]: util.py[WARNING]: Running scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python2.6/site-packages/cloudinit/config/cc_scripts_user.pyc'>) failed
Apr 20 13:15:29 cloud-init[1329]: util.py[DEBUG]: Running scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python2.6/site-packages/cloudinit/config/cc_scripts_user.pyc'>) failed
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/cloudinit/stages.py", line 553, in _run_modules
    cc.run(run_name, mod.handle, func_args, freq=freq)
  File "/usr/lib/python2.6/site-packages/cloudinit/cloud.py", line 63, in run
    return self._runners.run(name, functor, args, freq, clear_on_fail)
  File "/usr/lib/python2.6/site-packages/cloudinit/helpers.py", line 197, in run
    results = functor(*args)
  File "/usr/lib/python2.6/site-packages/cloudinit/config/cc_scripts_user.py", line 38, in handle
    util.runparts(runparts_path)
  File "/usr/lib/python2.6/site-packages/cloudinit/util.py", line 644, in runparts
    % (len(failed), len(attempted)))
RuntimeError: Runparts: 1 failures in 1 attempted commands
Apr 20 13:15:29 cloud-init[1329]: util.py[DEBUG]: Writing to /var/lib/cloud/instances/i-02ebcbe5/sem/config_ssh_authkey_fingerprints - wb: [420] 20 bytes
Apr 20 13:15:29 cloud-init[1329]: helpers.py[DEBUG]: Running config-ssh-authkey-fingerprints using lock (<FileLock using file '/var/lib/cloud/instances/i-02ebcbe5/sem/config_ssh_authkey_fingerprints'>)

Thanks :)

@brettgpalmer

I'm having similar problems. There appears to be a problem with the 'sudo' configuration for this instance. I was only able to get dynamic-dynamodb installed using the above mentioned command:

sudo /usr/local/bin/pip install -U -r /etc/dynamic-dynamodb/requirements.txt

If I try to run sudo pip it can't find it in the path.

Also, after I get the configuration setup I wasn't able to use the "service" command to start and stop the service.

sudo service dynamic-dynamodb start
-- returns without an error but no processes are started.

For a work around I used the following command to run dynamic-dynamodb:

dynamic-dynamodb -c /etc/dynamic-dynamodb/dynamic-dynamodb.conf &

This appears to be working for me now, but any hints on how to get service working would be appreciated.

Thanks

@jasonrdsouza
Contributor

I ran into this issue today as well. I created PR #242 to fix it, but to summarize, as @coolbox pointed out, invoking pip via its absolute path (/usr/local/bin/pip) should work to get dynamic-dynamodb installed.

To get the service aspect working, and thereby fix the issue of dynamic-dynamodb not starting up after an ec2 instance termination, you have to modify the init.d file that gets written by the cloudformation template to point to the correct location of the dynamic-dynamodb binary. Since pip is installed in /usr/local/bin, the dynamic-dynamodb binary is also in /usr/local/bin, but the init.d file incorrectly points to /usr/bin instead. Changing that should get everything working correctly.

@sebdah
Owner
sebdah commented Apr 25, 2015

Thanks for the pull request @jasonrdsouza. It has now been merged and released in version 1.20.5!

@sebdah sebdah added this to the 1.20.x milestone Apr 25, 2015
@sebdah sebdah self-assigned this Apr 25, 2015
@sebdah sebdah closed this Apr 25, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment