Skip to content
This repository has been archived by the owner on Jan 31, 2019. It is now read-only.

[bug 1173085] deal with ebs mount edge case #170

Merged
merged 1 commit into from Jun 11, 2015
Merged

[bug 1173085] deal with ebs mount edge case #170

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
45 changes: 36 additions & 9 deletions puppet/modules/socorro/manifests/role/processor.pp
View file
Open in desktop
@@ -1,28 +1,55 @@
# Set up a processor node.
class socorro::role::processor {

include socorro::role::common
include socorro::role::common

# Symbols uses a hierarchy of directories that should be on the same
# block device for performance reasons.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this comment is true enough, I just wanted to point out that if you don't put these on the same device stackwalker will totally fail to cache symbols at all (since the rename() will fail).

It's true that the reason it does rename() instead of a "copy-and-delete" is for perf reasons though, so true enough :)

$symbols_base = '/mnt/symbols'
$symbols_dirs = [
"${symbols_base}/cache",
"${symbols_base}/tmp"
]

file {
$symbols_base:
ensure => directory
}

# There is an edge-case interaction between cloud-init and *some* instance
# types that causes EBS volumes to be pre-mounted.
# https://bugzilla.mozilla.org/show_bug.cgi?id=1173085
exec {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a simple and decidedly inelegant solution: if the mount is there, unmount it. We can get away with this because we're using a strict role separation model; however, if we ever intend to combine roles, this may be problematic down the line.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what you mean by combine roles, like apply multiple roles to the same node?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean if we combine two roles that both expect exclusive use of /dev/xvdb. Unlikely, but wanted to point it out anyway.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah good point - yes I would definitely like to handle ephemeral storage in a more general way. I think this is fine for the moment since it's only really critical for processor.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ES data nodes also mount this device, fwiw...

'format-symbol-cache':
'check-premounted-ebs':
path => '/bin',
command => 'umount /dev/xvdb',
onlyif => 'mount | grep xvdb'
}

exec {
'format-symbols-cache':
path => '/usr/sbin',
command => 'mkfs.ext4 /dev/xvdc'
command => 'mkfs.ext4 /dev/xvdb',
require => Exec['check-premounted-ebs']
}

mount {
'/mnt':
$symbols_base:
ensure => mounted,
device => '/dev/xvdc',
device => '/dev/xvdb',
fstype => 'ext4',
options => 'defaults',
require => Exec['format-symbol-cache']
require => [
Exec['format-symbols-cache'],
File[$symbols_base]
]
}

file {
'/mnt/symbolcache':
$symbols_dirs:
ensure => directory,
owner => 'socorro',
require => Mount['/mnt']
require => Mount[$symbols_base]
}

service {
Expand All @@ -31,7 +58,7 @@
enable => true,
require => [
Exec['join_consul_cluster'],
File['/mnt/symbolcache']
File[$symbols_dirs]
]
}

Expand Down
21 changes: 10 additions & 11 deletions terraform/processor/main.tf
View file
Open in desktop
Expand Up @@ -71,9 +71,8 @@ resource "aws_launch_configuration" "lc-processor" {
lifecycle {
create_before_destroy = true
}

ephemeral_block_device {
device_name = "/dev/xvdc"
device_name = "/dev/xvdb"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I set this to b since it's the second device; if it needs to be c we can switch back.

virtual_name = "ephemeral0"
}
}
Expand All @@ -94,18 +93,18 @@ resource "aws_autoscaling_group" "as-processor" {
min_size = "${lookup(var.processor_num, var.environment)}"
desired_capacity = "${lookup(var.processor_num, var.environment)}"
tag {
key = "Environment"
value = "${var.environment}"
propagate_at_launch = true
key = "Environment"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bonus: I fixed some whitespace issues. 馃槒

value = "${var.environment}"
propagate_at_launch = true
}
tag {
key = "role"
value = "processor"
propagate_at_launch = true
key = "role"
value = "processor"
propagate_at_launch = true
}
tag {
key = "project"
value = "socorro"
propagate_at_launch = true
key = "project"
value = "socorro"
propagate_at_launch = true
}
}