deploy from local working copy rather than checking everything out from github

[glyph]

If an administrator has a complete local checkout, they should be able to deploy everything from that checkout.

For one thing, this makes it possible to deploy if Github is down or inaccessible from a given location. For another, it makes it possible to create test environments with commits that are not yet on github.

I've done a little bit of this in 129caab but have only put it into t-web. It needs to be done to the rest of the services.

[adiroiban]

The other part of the commit is twisted-infra/t-web@d1c1a93 :)

So I need to create a pull request for each of the submodules to change from branch into push and then come back on braid and update the submodules revisions.

---

A stupid question: Why not have all infrastructure code into a single repo? ... following GitHub Issues is already a mess for me.

---

It might be another stupid question, but why not use chef/salt for infrastructure.

Are they that bad so that Twisted needs its own tool?

With chef/salt it should be much easier to have a sysadmin work on this as there is no longer the need to learn how to use braid.

And it looks like braid is designed for developers and not sysadmins :)

[glyph]

I didn't make the decisions about submodules for infrastructure or fabric rather than chef; you'd have to ask @tomprince . I did voice a preference for something in Python just so we could maintain it with at least some of the skills our developers already have. Keep in mind, we do not have sysadmins, we only have developers, so solving problems a developer way is leveraging the resources we have. That said, Braid is not developer-friendly because you can't test it offline, which is exactly what this ticket is about ;).

Braid isn't really "its own tool"; if you use chef, you write cookbooks, if you use Ansible, you write playbooks, if you use Fabric, you write commands. Braid is just Twisted's fabric commands.

One reason we did go with Fabric rather than one of these other tools is that most devops tools are oriented around lots of immutable infrastructure which all needs to be provisioned around a single central database. In our case, almost every piece of infrastructure has mutable portions; the buildmaster has its build set, trac has its attachments filesystem, postgres is obviously a database, and the version control system has the repo. So we more needed something for automating migrating blobs of existing mutable data around than just provisioning new infrastructure.

[glyph]

Fundamentally though the reason things are the way that they are is that they are that way. There are some pieces of the infrastructure which have to be in separate repositories (highscores for licensing reasons, there's a private submodule which contains credentials for cloud services and buildbot, and so on)

[glyph]

So if we are going to have a discussion about re-architecting everything from scratch, I'd prefer to do it after we have a test infrastructure where we can test that change, which means #75 should be addressed :). So let's do that first.

[adiroiban]

Thanks for the info. I have already start working on #75 ... see #77

I understand the need for a private submodule (I am using a similar design for my project) ... but each up in a submodule master will need to be paired with an update in braid master... and I don't see what we gain.

After #75 is closed we can talk about merging submodules into braid.

[adiroiban]

We are working to move all submodules as local folder and with this we no longer need to work on this ticket.

[glyph]

@adiroiban - I don't think that is the case. Right now the local submodules are checked out individually from github at the locations where they're deployed. All that smashing the submodules into a single repository will accomplish will be putting more code into each location (for example, deploying all of the twisted web configuration on the buildbot, all of buildbot's configuration on trac, etc). There also needs to be a change to deploy code from the working copy after pushing it rather than deploying code from github.

[adiroiban]

@glyph the services/buildbot/fabfile.py only copies the files from services/buildbot into the remote buildmaster. similar to the previous git clone command.

Here is the code https://github.com/twisted-infra/braid/blob/master/services/buildbot/fabfile.py#L65 ... it only copies files based on os.path.dirname(file)

the code is deployed (in staging and production) from the working copy... not github.

[glyph]

Just for buildbot or for all services? This was supposed to be a ticket tracking the migration of all services to deploy via working copy.

[adiroiban]

Amber and I have submitted PR for all services... some were already merged...see https://github.com/twisted-infra/braid/tree/master/services

mailman and codespeed are still waiting to be reviewed.

for diffresource I have opened a ticket and which I argue that we can remove it :)

[glyph]

Cool :-). Please go ahead and link those to this one, then!

[hawkowl]

I think we can put this as done.