Skip to content

Commit

Permalink
Keyword 'roles' is not required for fs_templates so don't assume it e…
Browse files Browse the repository at this point in the history 
…xists
  • Loading branch information
@afgane
afgane committed Jul 2, 2015
1 parent 337ecb9 commit b56a6cd
Showing 1 changed file with 6 additions and 2 deletions.
8 changes: 6 additions & 2 deletions cm/master.py
View file
Expand Up @@ -310,9 +310,13 @@ def load_legacy_snapshot_data(self):

@TestFlag(10)
def get_default_data_size(self):
"""
Inspect file system templates provided in the config to figure out the
default size of the ``galaxy file system``.
"""
if not self.default_galaxy_data_size:
for fs_template in self.app.config.filesystem_templates:
roles = ServiceRole.from_string_array(fs_template['roles'])
roles = ServiceRole.from_string_array(fs_template.get('roles', []))
if ServiceRole.GALAXY_DATA in roles:
if 'size' in fs_template:
self.default_galaxy_data_size = fs_template['size']
Expand All @@ -325,7 +329,7 @@ def get_default_data_size(self):
log.warning("Could not get snapshot {0} size (setting the "
"value to 10): {1}".format(fs_template['snap_id'], e))
self.default_galaxy_data_size = 10
log.debug("Got default galaxy FS size as {0}GB".format(
log.debug("Default galaxy FS data size set to {0}GB".format(
self.default_galaxy_data_size))
return str(self.default_galaxy_data_size)

Expand Down

14 comments on commit b56a6cd

@kaktus42
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to add a FS with this, but it didn't work.
Unfortunately I don't have an errror message for you.

 - name: data
   snap_id: snap-52c06325

@kaktus42
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

push @afgane - how to add a volume so it will be accessible in the whole cluster?

@afgane
Copy link
Contributor Author

@afgane afgane commented on b56a6cd Jul 13, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm going to need some error logs or something to go by because I've tested this twice now on different accounts and it's working. Are you sure you're running source tip from the CloudMan repo?

@kaktus42
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I use https://launch.usegalaxy.org with the default bucket (cludman-test I think) - is that the tip?

@afgane
Copy link
Contributor Author

@afgane afgane commented on b56a6cd Jul 14, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No - cloudman-test is about 3 weeks behind tip now and contains the RC1 code. cloudman-dev bucket will (most of the time) have the tip (or the development version not yet committed) but it can also (& often) have broken code there as well.

I had sent you the instructions and scripts to easily create your own bucket with any modifications so I thought you were using that. I've since then added that bit of documentation here as well: https://wiki.galaxyproject.org/CloudMan/CustomizeGalaxyCloud#Using_custom_CloudMan_application

@kaktus42
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. I got the scripts. I need to focus on the data analysis more now and try to find a stable way for my analysis environment. I'm trying to balance new features with stability and decided to wait with galaxy on dev until the new release in a few weeks.
For the mounting of another drive, I will try to use cloudman-dev for now. Can I also attach and mount an existing EBS instead of a snapshot?

@afgane
Copy link
Contributor Author

@afgane afgane commented on b56a6cd Jul 14, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me try the volume option (never did) and will let you know.
(currently fixing your other bug report about volumes larger than 1GB)

@afgane
Copy link
Contributor Author

@afgane afgane commented on b56a6cd Jul 14, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automatically adding volumes at start should be possible with this commit 093f310.

As the comment in the commit says, when launching an instance, make sure to set the instance placement to be the same as where the volume resides.

Let me know how it goes.

@kaktus42
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried following settings with this outcome (don't know if it's the same, because I don't have another log file)

default bucket filesystem_templates
cloudman-dev snap_id: snap-860915f6
galaxy-cip snap_id: snap-860915f6
galaxy-cip volume_id: vol-fb635132

galaxy-cip has cm.tar.gz with the master tip from yesterday and the cm_boot.py from the same.
volume vol-fb635132 is in the correct placement.

2015-07-15 14:42:20,716 DEBUG  cm_boot:428 - virtual-burrito seems to be installed
2015-07-15 14:42:24,361 DEBUG  cm_boot:25  - Successfully ran '/bin/bash -l -c 'VIRTUALENVWRAPPER_LOG_DIR=/tmp/; HOME=/home/ubuntu; . /home/ubuntu/.venvburrito/startup.sh; lsvirtualenv | grep CM''
2015-07-15 14:42:24,362 DEBUG  cm_boot:431 - 'CM' virtualenv found

After that, nothing seems to change. I found out that the next step in the log file should be that nginx is beeing started?

@afgane
Copy link
Contributor Author

@afgane afgane commented on b56a6cd Jul 15, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you remember which AMI was used? Try with ami-c1ad79aa (RC3; now the default on launch.usegalaxy.org with an update to cloudman-test bucket to yesterday's source tip). Some of the changes over the past week required a new AMI so it's likely that. The one-up-in-the-log-chain file to look at would be /usr/bin/ec2autorun.log and see if there's an error message captured there.

@kaktus42
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ay. Found the problem to my email.
the RC3 ami (ami-c1ad79aa) is not public

And in the ec2autorun.log was the problem, that it seems you changed the paths where to find cm_boot*.
in /opt or in /tmp

@afgane
Copy link
Contributor Author

@afgane afgane commented on b56a6cd Jul 15, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry about the AMI - a bit rushed to wrap this up so didn't go through the checklist yet... It's public now.

Yes, the new path is under /opt/cloudman (vs. /tmp/cm) - this move is part of a larger security hardening effort so a fairly big set of changes for this late in the release process but needed to be done.

@kaktus42
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works with RC3 now. volume could be mounted and the processes on other machines can apperently acces the data on the other volume 👍 Thanks a lot!

@afgane
Copy link
Contributor Author

@afgane afgane commented on b56a6cd Jul 15, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Glad to hear it's all come together finally. Let me know how it progresses and if you run up against more features you'd like to see.

Please sign in to comment.