"Error: Remote servers are not synced."

[thelinuxlich]

What am I supposed to do in this case?

[gregberge]

@thelinuxlich cleaning the remote servers to have the same releases on it

[thelinuxlich]

Shouldn't the shipit-deploy have a option to do this automatically so we could trust the automatic deployment?

[gregberge]

@thelinuxlich yes maybe, it's just to prevent rollback to break the remotes

[thelinuxlich]

In a cluster environment, you want every instance to be identical, so this should be an option

[dmkelly]

As an example, what would happen if you added a third instance to a cluster environment? In that case, rollbacks won't work, but you would still want to deploy to it.

[twalling]

I would argue that this enforcement or check for synced releases should be done on the rollback task. People are going to want to rotate in or spin up new instances so even though its ideal, not all instances will have the same releases. It shouldn't prevent one from deploying though, it should prevent you from rolling back. Basically you can only rollback to the version that every instance has a copy of.

[thelinuxlich]

totally agree

[sharathprabhal]

@neoziro This limits us from rolling new instances into the cluster. +1 for making this optional.

[brycefisher]

@neoziro I also feel that not addressing this issue makes shipit very unfriendly to scaling a cluster, and it seems there's definitely some interest in moving this direction from others.

Seems there's several options to proceed on this issue:

1. Do nothing -- adding new instances to a cluster cannot be automated as part of shipit
2. Add a config flag to enable the sync check, defaulting to on -- this works for everyone and breaks no one
3. Move the sync check to the rollback -- this a big departure from current implementation but might be logical
4. Break the sync check into a separate task in shipit -- gives the project more control over how shipit works, but a major hassle

Of these options, (2) seems the best compromise:

• Smallest change to the deployment logic
• Least code to write
• Maintains existing deployment logic for users who don't want this change
• Enables clustered environments to disable the synced releases check

@neoziro if you agree that (2) is the best option, I'd be happy to submit a PR (and possibly tests) for thist too.

[gregberge]

As I understand, if you have these problem, this is because you don't deploy all servers in one shipit isn't it? So if you are doing that, you should proceed exactly the same way to rollback your servers.

[brycefisher]

@neoziro Here's how I reproduced this same issue:

1. Provision a new server (ex: launch a new ec2 instance)
2. Update shipit config to include the new server
3. Run the shipit deployment

This causes the deployment to fail because the new server (from step 1) does not have any releases, whereas preexisting servers do have releases.

There's several good reasons to provision new servers:

• scale up in anticipation of a PR announcement
• scale up in response to a traffic spike
• patch the configuration of the OS in cloud environments (security updates or other changes)
• migration to new physical infrastructure in a dedicated hosting or on-premises data center

The requirement to have matching releases means that I can't use shipit until I manually synchronize the names and numbers of folders, even if I need to run an urgent deployment for a traffic spike or security update. It breaks down the automation in a significant way for cloud environments.

By making the synchronization of previous releases a config setting (turned ON by default), existing consumers of shipit will keep the assurance they can rollback to a consistent state.

Consumers (like myself) who prioritize the ability to automate launching and deploying to new instances over consistent rollback can make that choice for themselves. By turning off the check for sychronized releases, consumers give up consistent rollbacks for a period of time. However, after the number of successful deployment is > number of old releases stored on disk, consistent rollbacks are no longer an issue.

IMHO, this seems like a win for everyone.

[gregberge]

@brycefisher I understand the problem, it's when you add a new server to an existing cluster.. OK in this case the option can be interesting, but it still very dangerous, because if one day your servers are not synced, the rollback can break everything.

I think that to compare we must use the REVISION file, it's the only way to know if the rollback will be synced or not.

If someone want to do a PR on it I will be happy to review it, else I will do it when I will have time.

[gregberge]

I moved it to shipit-deploy, shipitjs/shipit-deploy#70

[bpinter]

Hi there. So, is there a way to turn off the sync check? If i would like to add new instance to the previous ones? Because currently it breaks, so I can't be done unless I cleanup the old servers, which isn't very deployment friendly.

[gregberge]

@bpinter instead of cleaning up old servers, just install the new server like the old ones.

[bpinter]

@neoziro The old ones have the revisions already, but not the new, fresh one. So the deploy will fail, because of servers are not sync. The build is done on a jenkins server, the shipit only rsync the code to the app servers in my case.

[Lakret]

@neoziro Bumped into the same problem. Right now the only solution that's working is to manually copy old releases, and create a symlink on a new server. Doesn't look very "automatic" :)

[bpinter]

I'm thinking on the same as @brycefisher:
"By making the synchronization of previous releases a config setting (turned ON by default), existing consumers of shipit will keep the assurance they can rollback to a consistent state."

[brycefisher]

After having worked on this for several days, I realized that the change @neoziro is asking for amounts to rewriting a significant portion of shipit, possibly the majority of the module. As @bpinter and I agree, the most efficient and mutually beneficial way to proceed is to add a configuration off to entirely disable rollback / synchronization. For my team, any rollbacks we do are done by reverting a commit in git, not by using shipit.

[thelinuxlich]

It is sad that this and other important issues still remains, seems like the project stagnated :(

[bpinter]

Hi there. Guys, are you planning changes in the near future to change the behaviour and make the (btw. very cool) system more 'automatic deployment' friendly? Thank you.

[brycefisher]

@bpinter I think I'm out of time to work on this personally. Not sure what @neoziro has in store.

[Tomtomgo]

+1, would be very good to have!

[gregberge]

@thelinuxlich yes I don't have so much time right now to take care of these important issues.

Yes, this issue must be addressed, I reopen it.

[gregberge]

Already opened in shipit-deploy

[alexander37137]

I created small pull request that add option to disable release check shipitjs/shipit-deploy#162