reorganize the repos ?

[ldemailly]

Couple of options to get us started; more welcome:

a) move everything into a mono repo so we can share code instead of duplicate it, have a unified development experience and not spend too much time moving PRs from 1 repo to the next

b) add more repos so we can actually share code instead of duplicate code between pilot and mixer for instance (istio/core ?) and istio/generated for generated code sync and istio/build for flat build and use the "repo" tool to keep things in sync

c) [not an option IMO] no change

cc @costinm

[kyessenov]

b) is already done by test-infra and api repos. They're imported into each project.
Are we trying to address developer pains here, CI pains, or release pains?
To make local development easy, I think using local_repository from bazel is probably the simplest multi-project bazel solution. It is in accordance with standard go tooling as well with the canonical code paths.

[ldemailly]

Ideally we'd find a solution addressing most/all of our pains at once

Api repo maybe can be renamed to "core" because it seems odd to put code or utility libraries into a repo named "api" (and I think for generated it'd be best for bots keeping generated code updated to not fight with humans PRs)

The short version is either we do a really good job at our dependency graphs and not hesitate to add more light weight repos (which would inherit shared build tooling) when a new node is needed in the graph (eg minimal mixer for testing in pilot, shared between mixer and pilot (and proxy)) or we push all code to the lowest repo which becomes mono repo

or some other creative option

[geeknoid]

As time goes on, I think we'll have more and more stuff we want to share between Pilot, Mixer, CA, and Broker. We definitely need a spot to share utility libraries across the project. Additionally, we want to be able to have a unified tooling experience (linters, etc). And finally, as we put together a perf testing/tracking framework, we'll want to share that between components.

I'm not seeing the inherent value we're getting out of having distinct repos at this point. It seems to be mostly introducing pain and providing very little upside. I would be in favor of consolidating into a single repo for most of our code with different subdirectories for each major component.

[jmuk]

A single monolithic repository of sharing code among istio components would be great for us -- "us" meaning istio devs. I'm wondering if this can affect the workflow of istio users (operators).

I mean, istio operators need the yaml files in install/kubernetes in istio/istio repository for setting up istio in their cluster. It is unclear to me where do those files go; in the same repository (so operators need the large repository for setting up istio), or do we keep istio/istio repository for the setup? (or do we make a different non-git-repository location for the official setup files?) This might also affect the e2e code structure.

[ldemailly]

@jmuk the yaml files are part of the release artifacts (for instance in https://github.com/istio/istio/releases/download/0.2.6/istio-0.2.6-osx.tar.gz or what curl -L https://git.io/getLatestIstio | sh - automatically downloads),
operators don't need to git clone the source tree so it shouldn't be a concern either way here

ps: and our git clone of istio/istio is 80Mb because it had some demo/binary artifacts accidentally checked in at some point in the past (#459) so if we were to clean that it's likely the total source tree in a new clean mono repo would be much smaller than the current partial one anyway.

[jmuk]

okay, thanks for the clarification.

[Brian-Xincheng-Zhang]

if we can keep a single repo, can we still make individual release for different components, say pilot, mixer
If so, I support single repo.

[ldemailly]

the source code being in one place doesn't preclude from making different artefact under different schedules if one desires so. I think it's already hard enough to make 1 unified release and would be premature to decouple the components at this point though (but that's a somewhat different topic)

[Brian-Xincheng-Zhang]

component based release does not have to do it now, but probably a 2018 goal.
as long as it is doable, then it is fine.
Laurent: could you discuss this one repo proposal in tomorrow's working group meeting?

[ldemailly]

Please give comments in this doc: https://goo.gl/eLq2uS

[ayj]

kubernetes/kubernetes#24343 lists reasons Kubernetes started to move away from a monorepo.

[kyessenov]

@ayj that's a great thread. I think mono-repo is a crude tool to solve some of our problems, that may also introduce other issues down the road. Perhaps, we should focus the discussion on what's exactly painful and try to address them in the specific contexts.

[wattli]

Does this problem only happen for pilot and mixer or there are other places?

[ijsnellf]

This discussion has come up on every project I've worked on. Unfortunately, I think both methods have plenty of issues. The only thing I'll put in is that switching from one to the other is painful, and the grass is always greener on the other side.

[elevran]

Is discussion happening on this issue or https://goo.gl/eLq2uS?
How will a decision be reached and communicated?

[costinm]

Please also check: https://docs.google.com/document/d/1u8kn6_R6oScmj498kETFj0g0f3Qld2vq7xYi0E2DgJI/edit# Note that the 2 POCs are pretty basic - and include various other experiments (better IDE support, faster initial builds, etc), and are quite incomplete - for example the single-project proxy only replaces SHAs with 'local' for a couple of critical deps. I think most important is to gather feedback and votes from the community - since the main purpose is to improve developer experience and productivity. Costin

…

On Mon, Oct 9, 2017 at 7:05 AM, Etai Lev Ran ***@***.***> wrote: Is discussion happening on this issue or https://goo.gl/eLq2uS? How will a decision be reached and communicated? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#998 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAFI6mJMrirEgbD7Tk3mF-2_eVGThuEcks5sqid9gaJpZM4PpZDF> .

[sebastienvas]

Please also check: https://goo.gl/pzCCf4. There are a lot of improvement we can make with the current state of things. @costinm we should consolidate, or better split depending on which objective we want to accomplish.

[sdboyer]

hi folks - it may be a bit orthogonal to your thinking here, but at least some of what you're grappling with here may be addressed by introducing first-class language-level dependency management via https://github.com/golang/dep. if that's something y'all want to look into, i'm happy to answer questions or provide guidance, as needed.

[costinm]

It's not orthogonal - how we organize dependencies is a major issue. We do have (experimental) dep in Istio pilot - unfortunately mixer is trickier, dep can't seem to resolve. The real issue is that Istio doesn't check in generated artifacts, and dep has no option to skip/ignore missing deps. In pilot we 'hack' it by having a separate git with generate artifacts (but it's not yet automated). If we could get the same thing for mixer, dep would be a viable solution for managing all go dependencies in Istio. Another issue is that dep doesn't appear to have an 'exclude' list, so projects already in gopath are not added to vendor (but workaround is easy).

…

On Sun, Oct 15, 2017 at 10:48 PM, sam boyer ***@***.***> wrote: hi folks - it may be a bit orthogonal to your thinking here, but at least some of what you're grappling with here may be addressed by introducing first-class language-level dependency management via https://github.com/golang/dep. if that's something y'all want to look into, i'm happy to answer questions or provide guidance, as needed. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#998 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAFI6oUEdxWC87jihKv_uGJ7PwuGWBvWks5ssu4UgaJpZM4PpZDF> .

[sdboyer]

Istio doesn't check in generated artifacts

generated artifacts, e.g., generated Go code, right? if so, yeah, that can definitely be a pain point. this was a calculated risk that we took in the design, based on guidance from the original blog post on go generate. however, there are some workarounds - golang/dep#1077 has more info and discussion.

dep has no option to skip/ignore missing deps.

yeah, it can't adaptively do this - dep's entire model is based around the idea of keeping things in sync. however...

Another issue is that dep doesn't appear to have an 'exclude' list, so projects already in gopath are not added to vendor

have you checked out the ignored list, perhaps? that also may help with your skip/ignore goal.

but yes, in general, dep is not designed for splitting responsibility between GOPATH and vendor. this can be painful for projects that have a cluster of repos folks are working on simultaneously, which i imagine may be the case for istio. i have some ideas about how to do it in the long term, once we're in the toolchain, as well as some possibilities for the medium term. the medium term stuff will have to wait until after golang/dep#121, though - though that's on a pretty near horizon, now.

[ldemailly]

This is mostly done - many thanks @mandarjog

Still left :

• the api->mixerclient->proxy->istio chain
• istio.io website

[ldemailly]

I'm going to close this one and leave the 2 open issues for later