Migrate from gost to gb

[xiam]

Our gost tool is currently receiving no attention. In the other hand gb is getting a lot of traction so it sounds like a good option for us.

See http://getgb.io/examples/getting-started/ and
https://www.youtube.com/watch?v=c3dW80eO88I

[uaalto]

This might be a good issue to get my hands into, even though is P1, is not actually touching end-user functionality and will help me get more familiar with the different sub-projects. What do you think?

[xiam]

Hey @uaalto,

I think it would be awesome to have this sorted out! I believe we have some repositories that we don't really need so I think it would be better to try to to build github.com/getlantern/flashlight from scratch and add repos as they're needed. Please be aware that some of the repositories mirrored under src may actually differ from their upstream repositories, it will be super useful to identify them so we can fix upstream too.

Thanks,

[uaalto]

Roger that! I'll take care, and get back to you if any question arises.

[aranhoide]

Note also that some repos may not be needed by flashlight, but we want them
in sync with the flashlight project. That is the case for the peerscanner
project at least. Still, I think @xiam's approach is sound to get the port
started. We can always add whatever projects we miss later.
O 5 xuñ, 2015 18:13, "Ulysses Aalto" notifications@github.com escribiu:

Roger that! I'll take care, and get back to you if any question arises.

—
Reply to this email directly or view it on GitHub
#2595 (comment)
.

[myleshorton]

Yeah to echo @xiam's point, at this point the canonical version of those repositories are the ones in https://github.com/getlantern/lantern. To be super clear, for example:

https://github.com/getlantern/lantern/tree/valencia/src/github.com/getlantern/fronted

is much more up-to-date than:

https://github.com/getlantern/fronted

Probably the way to go would be to first push all changes like that upstream (see https://github.com/getlantern/gost#contribute-changes-back-upstream) and to them start from scratch building flashlight alone, theoretically from flashlight itself with its changes pushed upstream, so from https://github.com/getlantern/flashlight, which is very out of date currently.

Does that make sense? It's basically a fairly significant project to undertake =) @uaalto, but have at it if you like! Just getting all the upstream versions updated and compiling would be significant step.

[uaalto]

Ok, I have everything lined up to start the migration. There is just one issue that I wanted to checkout with you guys.

The objective of vendoring is to have reproducible builds (i.e. exact versioning of all dependencies, one way being bundling copies in the same respository). That seems to be addressed by Gb. What is not so clear to me is that this tool helps solving the upstream synchronization problem. That is, 'gb vendor' plugin (which is actually out of the core scope of Gb) will fetch deps, but won't take care of updating upstream. On the other hand, Gost has this functionality to update upstream repositories.

The problem is that once Gb migration is in place, there won't be an easy way to update upstream repositories without the proper strategy or support. In other words, with Gb you get reproducible builds but at the cost of no default means of updating upstream repositories. Possible solutions would be:

1. A Gb plugin
2. A script using git subtrees or submodules
3. Manually using git subtrees or submodules

What do you think about this? I might be missing something!

[fffw]

I think the ability to push back changes to upstream is important for us,
though not a strong requirement for a build tool in general.

gb requires us to manually place all dependencies to a separate vendor
folder, even when it's another repo written by ourselves, which is a bit
counterintuitive.

On Mon, Jun 8, 2015 at 10:03 PM, Ulysses Aalto notifications@github.com
wrote:

Ok, I have everything lined up to start the migration. There is just one
issue that I wanted to checkout with you guys.

The objective of vendoring is to have reproducible builds (i.e. exact
versioning of all dependencies, one way being bundling them in the same
respository). That seems to be addressed by Gb. What is not so clear to me
is that this tool helps solving the upstream synchronization problem. That
is, 'gb vendor' plugin (which is actually out of the core scope of Gb) will
fetch deps, but won't take care of updating upstream. On the other hand,
Gost has this functionality to update upstream repositories.

The problem is that once Gb migration is in place, there won't be an easy
way to update upstream repositories without the proper strategy or support.
In other words, with Gb you get reproducible builds but at the cost of no
default means of updating upstream repositories. Possible solutions would
be:

1. A Gb plugin
2. A script using git subtrees or submodules
3. Manually using git subtrees or submodules

What do you think about this? I might be missing something!

—
Reply to this email directly or view it on GitHub
#2595 (comment)
.

BR,
Merlin

[xiam]

I agree with @fffw, I think it will require us a bit more effort to push packages to upstream but it won't be impossible. Manually using git submodules sounds to me like an option we can try, if we see this task is repetitive and can be scripted then we could create an script for that.

[uaalto]

Yes, I agree that implementing this functionality is better after we've gained some experience. Not implement it until it's needed (YAGNI).
There is a remaining point of discussion: submodules vs subtrees.
Basically, it boils down to (in our use case, as the list is extensive):

Submodules:

• Pros: Dependencies are easily kept in sync. The host project always knows when the submodule needs update or has uncommitted files.
• Cons: Updates must be done per project. You can push recursively, but you cannot (AFAIK) link all commits into one so it can be reviewed as a single commit.

Subtrees:

• Pros: Commits include modified files from all dependencies, which simplifies code review and makes commits more consistent.
• Cons: It's more difficult to keep in sync with upstream, and complicates management a little bit.

Gost was using Subtrees, and in my opinion for good reason. It is in any case what would change less how things are working now. Basically, we'd have a vendoring system (Gb) and we would handle subtrees manually (until and iff we see it adequate, we could try any of the options I mentioned earlier). Gost was trying to do both.
What do you think?

[uaalto]

I'd add also to the list of options to simplify pushing upstream the use of Git client-side hooks. This could be an easy way of automating upstream updates after a push. I need to see if is viable though.

[myleshorton]

Honestly @uaalto I would just start out with the vanilla approach of treating all repos outside our main repo as vendored from the main repo and not as submodules or subtrees but just as vendored code at a given snapshot. If we make changes and push them, we should be able to fairly easily script the update to the upstream repos after a push in a variety of ways, but I like your client-side hook idea.

[uaalto]

The process would be:

1. Update upstream repos
2. Rebuild flashlight project, in getlantern/flashlight
3. Make a copy of the dependencies via 'gb vendor', which will pull dependencies without any gitmodule/gitsubtree at all.

Is this correct?

I think you can just push changes upstream afterwards squashing commits anyway, using subtrees.