skilling_up_sw_devs_with_no_ops_experience
Wednesday, Greenhouse, 11:30
- Convener / Facilitator
- Note taker - John Fitzpatrick
- Matt
- Jon Cowie, Etsy
- Gavin Heavyside, MyDrive Solutions
- Gogs, Comparethemarket
- Alex Manly, Chef
- John Fitzpatrick, Chef
- Jason
- Mandi Walls, Chef
- Yasser
- Janil
- Denis, Betfair
- Olivier, Bla bla car
- Marco
Question: How do we skill up s/w developers in Ops?
How many people let devs make infra structure changes - 4 put up their hands!
Matt: Has 7-8 people in teach on Chef, the ops guy PITA having to give the basic intro on how chef/cookbooks work. Spends a lot of time with them until they're familiar - big time suck.
Jon: Should get other developers to the point where they can help out - shouldn't be responsibility of one person. They may be able to help out, but tehy just dont. E.g., had a PHP5.5 upgrade steered by developers, and others skilling up team members. Document the tools, e.g. foodcritic, so best practices are built in. E.g.
How much time spent is teaching generic stuff, and how much org specific?
Jon: "Enlightened self interest". Need to sell to them the tool. Best selling point is this will help you work quicker. With teams at his company that dont use chef as often, Jon would help them get started, but show they can do this themselves.
What are the blockers? Accountability may be an issue. Should have a blameless culture.
Depends on peoples backgrounds - if they worked in large org may not have seen production, but in small orgs they may have.
If you dont have a blameless culture, people will be afraid of making mistakes. "You will never be fired for taking down the site" sign on wall. Should treat problems as a learning exercise.
'Ops and allied trades' - what is it you're automating - maybe there is less requirements for training. For a front-end developers, there would be more of a handholding. Empower people, but don't drop them in at the deep end.
Regulatory compliance may be a concern, e.g. PCI compliance, should have cookbook review.
Gavin: TDD code should make people more comfortable - serverspec, etc.
Jon: "TDD only gives them confidence that they're test suites work". Use exceptional handlers to show when things fail - graphs, pipe out to IRC, etc. They test chef changes on a single server- use physical boxes, not virtualized.
Gogs: Implements more tests do give a lot of confidence in their org.
Marco: trying to training sys admins & developers. Developers are enthusiastic - versioning, reuse, etc, but Sys Admins dont want to learn and want to maintain things manually. 37yr old comp, so not startup, and hard to change peolpe. Old IT based company - just now using cloud. Developers are scared of administrating system - OK with writing cookbooks - for more than 30yrs someone else did it. Using chef as a common language between these two worlds.
AlexM - its about managing perceptions, maybe work with a small team, once they get things working, then use that to spread the word within the organisation. Introduce slowly, then show the benefits to the management.
Marco: Theoretically thats right, but they had to set up so many projects...
Jon: Always going to struggle w/ strict silos and hard to make changes. One thing to help is to appeal to 'mutual self-interest'. As a developer, when you deploy a feature its your responsibility, - should have a developers on call.
AlexM: Problems, in big orgs some developers are contracted in.
Developers may get called as well, just make them more responsible from the outset. Set up a culture of shared ownership.
But Contractor Developers are not responsible. With permanent staff, page developers at the same time as support staff. Change the mindset for developers and sys admin that its a shared responsibility.
Denis: They would always call the developer in when there is a issue. They'd also give developers access to production!
Management buy-in is key to organisation and cultural changes. Devs will always want to just hand stuff over the wall and walk out!
Jon - Once dev get into the way of being interrupted in middle of the night they will be come more receptive to helping from the outset.
AlexM - what about sys admin using same cookbooks as developer?
Jon - Its all about minimising friction to make changes and rampup. If you're doing in production you should want to replicate that in dev.
Denis uses Chef 9, so no environments, so has different repos. Jon uses environments,and there is no diff between dev, testing and production.
Jon - "Give any developers sudo access on production boxes, so they can e.g. check memory on db shards etc - so they can identify the problem before they go to a DBA with with relevant info
Should have domain experts on call - core platform, DBA, clustering issues, search/solr issues, etc. Like Developer on-call procedure, similar to SysAdmin on call procedure!