Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: make an "official" go-standards/project-layout documentation within Go project #45861

Open
idexter opened this issue Apr 29, 2021 · 17 comments

Comments

@idexter
Copy link

@idexter idexter commented Apr 29, 2021

Context

A couple of weeks ago @rsc created an issue golang-standards/project-layout#117 which started pretty hot discussion within the community.

Despite the fact I personally don't see any problems with that repository and I agree with repository author: golang-standards/project-layout#117 (comment) I want to help the community and Go team.

Why I don't see an issue with original repo

There a lot of organizations which can be misinterpreted and which starts with golang-.
Here are some examples:

Today most of them not as popular as https://github.com/golang-standards/project-layout and empty. But tomorrow they can become popular.

It's obvious that does not make any sense to restrict or try to close or rename them. It's also obvious that community have a request to get some guide and official view about structuring different type of projects which using Go. I think that one of the reason original repo became so popular.

I also understand that @rsc and some other top members probably want to make their live easier, as a maintainers. The fact some (probably a lot) newbies misunderstand the purpose of original repository makes me sad to. I always explain my teammates that that repo is not an official (It they don't read README and it's content carefully). Sometimes it's tedious process. Sometimes it's really annoying when some newbie came to pretty simple repository and make an PR trying implement project-layout as-is.

At the same time I understand @kcq and other developers who see an original repo useful and who contributed to it and used it in their projects. They also did a lot of work and I think we should respect everyone in the community.

What I propose

I propose create a repository or some article/document within official https://github.com/golang organization which will be used for describing common patterns regarding go projects structure and which will be moderated and discussed with Go team.

It can be something like:

Then we can get some ideas from https://github.com/golang-standards/project-layout which Go team agree and remove disputable things which Go team disagree. Or maybe make it from scratch. I don't know

I think it will help target community request to have authority source of truth about "official" project structure and at the same time it will help so save original repo and all work community has done within that repo.

Eventually official repo most likely became more popular than original one and there will be no need to fight about renaming it.

@henvic
Copy link
Contributor

@henvic henvic commented Apr 29, 2021

The problem with this, in my opinion, is it's not like the Go team doesn't have anything in this regard.

And many other internal pages, besides https://go.dev/, are exactly what such repo would do if it existed.

@idexter
Copy link
Author

@idexter idexter commented Apr 29, 2021

and many other internal pages, besides https://go.dev/ are exactly what such repo would do, if it existed.

It will consist of guidelines for the standard structure of various types of Go projects (libraries, cli applications, web applications, etc.) in one place on one page with an easy way to find it on golang.org, go.dev and even Google Search

At this point, the documents look fragmented and hard to find for some newbies. It may be time to restructure old documents. Perhaps the problem is that the existing documentation structure is confusing.

@henvic
Copy link
Contributor

@henvic henvic commented Apr 29, 2021

¯\_(ツ)_/¯

@ianlancetaylor ianlancetaylor changed the title Proposal: Make an "official" go-standards/project-layout documentation within Go project. proposal: make an "official" go-standards/project-layout documentation within Go project Apr 30, 2021
@ianlancetaylor ianlancetaylor added this to Incoming in Proposals Apr 30, 2021
@kylelemons
Copy link
Contributor

@kylelemons kylelemons commented Apr 30, 2021

I see this come up very frequently for new Go developers. They are looking for guidance, for the "right way to do things." As such, I am certainly in favor of doing something about this, but I don't know if making a new repo in the git org is the right approach, or if there's a one-size-fits-all solution.

TL;DR: I think we should have a wiki page that covers how to think about a few use-cases, and put on-ramps in the FAQ, Effective Go, getting started, tour, etc.

If I haven't lost you yet, here are some approximately-structured ideas on the subject, broken down by "train of thought:"

Should an "official" golang-standards repo be created?

Potential pros of a git repo in e.g. the golang org:

  • People can clone it (though they'd either have to remove .git or inherit its history)
  • People can fork it, use git blame, etc
  • SEO coattails

Cons of a git repo:

  • History is probably not that interesting for this.
  • There is no single correct layout, so copying whatever it has probably isn't what we want anyone to do
  • Confusion about where pull requests and issues go, since they'd presumably live here with everything else

I don't think the balance really favors making this a git repo.

What would the "standards" be?

Unfortunately, as I mentioned above, I don't think there can be a single standard. What a repo is going to look like should mirror how it should be used. Libraries need to look different than tools. Build ecosystem matters a ton. Project maturity and codebase matter a lot.

Off the top of my head, I can think of at least 4 "starter" layouts:

Simple library

A simple library that only has one primary, public surface, along with maybe some helper libraries that may or may not be internal. You run go get github.com/org_or_user/packagename and/or import "github.com/org_or_user/packagename"

README.md
LICENSE
go.mod
go.sum
packagename.go
packagename_test.go
internal/...

Simple tool

A simple tool that is intended to be installed, and not really imported. Might have some helper libraries too.

A simple library that only has one primary, public surface, along with maybe some helper libraries that may or may not be internal. You run go install github.com/org_or_user/toolname@version to install it.

README.md
LICENSE
go.mod
go.sum
main.go
main_test.go
internal/...

Repository with multiple commands or commands mixed with public libraries

Think something like kubernetes, protobuf, etc... something that has both an API consumer and a human CLI interface. Or just a repository that has both a client and a server. Utilizing the root of the repository in these cases may still be useful, but the clarity of at least one additional path element often seems warranted in these cases. To avoid using up import paths that make sense in code, often this is where I see the cmd prefix coming up.

README.md
LICENSE
go.mod
go.sum
cmd/tool1/main.go
cmd/tool2/main.go
packagename/packagename.go
packagename/packagename_test.go
internal/...

Repository with multiple languages or special build requirements

For example, a repository that has Bazel support. In these cases, there are lots of options based on what the structure and intent of the project will be. Many times these won't be installable or usable with the go tools at all, you may have to use Bazel if you want to include them in your project.

For a project that is providing a library for many languages, you might see something like this:

README.md
LICENSE
go.mod
go.sum
Makefile
go/packagename/packagename.go
go/packagename/packagename_test.go
go/internal/...
py/packagename/__init__.py

Or, it might have many languages simply because it made sense to implement parts of the project that way.

README.md
LICENSE
go.mod
go.sum
Makefile
static/...
frontend/Gemfile
backend/main.go
backend/packagename/packagename.go
backend/packagename/packagename_test.go
src/com/example/company/app/android/...
ios/project.xcodeworkspace

Where should guidance go, then?

We have a CodeReviewComments wiki entry and a SliceTricks wiki entry that I see very widely referenced. Wikis seem generally understood to be at least somewhat non-canonical, even though it's in the official repository, and we can have a least a little bit of nuance -- nobody will be rsyncing it to create the basis for a repository. That would give an opportunity to do something like what I outlined above: talk about how the way the repository is going to be used can influence package layout, and give some examples of simple, intermediate, and (probably by external link) complex or mature layouts.

I will note that there exists a GitHubCodeLayout wiki entry that I haven't seen referenced a lot, but does address some of this, and at least covers the "go get" concerns above. I can't say why this doesn't seem to fill this gap, but a few guesses: it might be a little outdated (though that's easy to fix) and that might lead to people not recommending it, or the on-ramps for it might be insufficient. It may also be that, despite my hypothesis, a wiki entry just doesn't have the SEO for this.

@sidh
Copy link

@sidh sidh commented Apr 30, 2021

Unfortunately, as I mentioned above, I don't think there can be a single standard. What a repo is going to look like should mirror how it should be used. Libraries need to look different than tools. Build ecosystem matters a ton. Project maturity and codebase matter a lot.

That is absolutely true. And layouts you provide are a good example of that. On the other hand this 'standard' or whatever its called does not have to solve all cases or encompass all uses. What new developers need is some guideline, a number of examples to pick from when they are starting their first project. They can change the layout when they get the general idea of how the things work. But these examples will be very useful when they begin.

Placing those layout examples under golang organization will carry a lot of weight. Layout examples is actually at better name than standard/whatever. Standard is too strong a word for what is actually needed.

@batara666
Copy link

@batara666 batara666 commented Apr 30, 2021

A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.

John Gall, Systemantics: How Systems Really Work and Especially How They Fail 1975

one reason i love Go Language:

Go syntax, stdlib, command line, ecosystem never cause me a difficulties and let me do what i want to do

I don't want to lost all the Go bliss just because of a "standard project structure"

like, why do I need to waste my time to follow the standard when i can just go with the x format, y format or just a flat format?

It's just waaay too far for taking other people business

Having Go Standard Project Structure will never as helpful as having a gofmt (In fact, It's may slow down, adding meaningless complexcity to our project)

or we can just polish up the Go Tour about "How to Structure Your Go Project" (Without mentioning the standard world or having official example)

@yordis
Copy link

@yordis yordis commented Apr 30, 2021

@batara666 your words are why I tried to explain in the long thread that we all, at the individual level, define the "go standards", too many invariants check in order to get it right (impossible).

That being said, we live in a world where some people need guidelines (the reason why the repo is so popular), so I would love the Go team to take some ownership, and add some "example layouts" with some basic explanation why for those who need it.

@batara666
Copy link

@batara666 batara666 commented Apr 30, 2021

@yordis

That being said, standard and official ownership is on the same level

and that's not the real problem

If your purpose are guiding novice how to structure their Go project, one dedicated Go Tour page is enough, not excessive

@nemith
Copy link
Contributor

@nemith nemith commented Apr 30, 2021

I think a well written blog post would go a long way vs a sample or template repository.

@powerman
Copy link

@powerman powerman commented May 1, 2021

I'd like @kylelemons examples above, but I think we should pay more attention to his last example. One of reasons why golang-standards/project-layout repo is so popular is provided guidelines about how to structure not-usual-Go-files: API, assets, scripts, templates, web static, etc. (some of these may contain Go files, but that's not important here). And while it's not-really-about-Go, it's still about "Go projects" - many Go projects contains such files and they have to decide how to structure them. Also there are multi-language projects/repo (and some languages already have own requirements/conventions how to structure their files), so it's nice to have an example of structure which allow at least most popular of them play nice together in same repo.

One may argue it's not Go team responsibility to decide how to structure web static or shell scripts. That's true. But people who write Go actually needs this, so as long as Go team avoids this topic we'll have a lot of 3rd-party blog posts and example repos like golang-standards/project-layout, most full of questionable quality advices and some of them even very popular no matter of their quality.

Probably this is more about more general question "how to structure complex multi-language backend service projects", which isn't Go-specific at all. But developing such types of projects in Go is very common, plus some other languages already provive guidelines or tools to solve this issue - that's why I think it's makes sense to provide a Go-specific answer on that question too.

Personally I believe decision "how to structure static, assets, templates, scripts" have exactly same nature as other decisions hardcoded in gofmt: it's not important at all, I don't like to decide this for each next project, I don't like to discuss it with my team - all I need is one recommended way to do this (ideally - used by everyone). For this only reason I'm currently in support for golang-standards/project-layout (only for complex backend projects, of course) - I don't agree with all of it, but I believe it's better to follow any popular enough structure than invent custom one on each next project.

@batara666
Copy link

@batara666 batara666 commented May 1, 2021

For this only reason I'm currently in support for golang-standards/project-layout (only for complex backend projects, of course) - I don't agree with all of it, but I believe it's better to follow any popular enough structure than invent custom one on each next project.

@powerman

  1. could you define what is complex backend projects? and what kind of project?
  2. Will someone shout at me when I don't follow moby/moby (60.1k stars) Project Structure for my qrcode project?

If the real purpose is giving guidance to novice how to structure their Go Project, I agree with @nemith solution (a well written blog post would go a long way vs a sample or template repository.)

@powerman
Copy link

@powerman powerman commented May 1, 2021

@batara666 To me it looks like your comment's goal is to turn this issue into same what happened with golang-standards/project-layout#117, so I'm not going to respond your 2nd question. Please keep this issue comments on-topic.

Complex backend project in the context of this discussion is one which contains enough both Go and non-Go files to make it too inconvenient to keep all non-Go files in project root and make it hard to distinguish root directories with Go packages and root directories containing something else. At this very moment project will needs some structure which differs from usual (shown in first 3 examples by @kylelemons comment above) Go project structure, for which we don't have good and officially-blessed example so far.

Official Go Blog post will works for me too, except it may be harder to update. In my experience project structure is evolving with time and new tools/frameworks may affect it, so it may need updates.

@codenoid
Copy link

@codenoid codenoid commented May 1, 2021

For anyone who just joined, let's recap the problem

  1. It starts when the repo golang-standards/project-layout indirectly claims that the repository is a standard

  1. majority request of renaming the "golang-standards" name, but there is no response from creator
  2. idexter creating a proposal proposal: make an "official" go-standards/project-layout documentation within Go project
@batara666
Copy link

@batara666 batara666 commented May 1, 2021

I understand why powerman feel that way, so a 3 .html and 2 .go file is the complex project.

sadly, There is never a language that explicitly define how a project dir/structure must be looks like.

  1. The famous Python, C, C#, JavaScript, PHP never have an Official Project Structure
  2. Java/Kotlin only giving tiny example, example: org.example.kotlin.network.socket should be in the network/socket subdirectory of the source root.

The solution for the Context of problem actually pretty simple:

  1. Rename the org
  2. Rewrite the Bio
  3. declare explicitly This is not official project structure of Go Language with H1
@idexter
Copy link
Author

@idexter idexter commented May 1, 2021

@codenoid @batara666 I explicitly described a problem as I see it. In my opinion the problem not so simple as you are trying to represent. I understand it's the simplest solution just force the author to change org name, or remove it. I also explained why it's bad idea it the proposal. Doing that you a trying to "solve the problem" of one part of the community without taking into account and infringing on the other part of the community. I suggested another way that will take into account the interests of all parties.

This proposal was made not for fights. Please read https://github.com/golang/proposal#the-proposal-process carefully.

The aim of this thread is:

Accept proposal, or decline proposal, or ask for a design doc.

If you don't like this proposal, it's not an issue, please create your own or go to the original repo and try to force author to change org name. But please stop fight. I don't want this topic became a mess.

Back to the topic

First of all thanks all for you thoughts. I like @kylelemons ideas as a starting point. I'm also not sure what is the best place for the recommendations.

The only things I know:

  • It should be at least reviewed by Go team so it can represent "official" point of view on how it could looks like.
  • This recommendations should not be described as requirement for all Go projects.
  • It should be pretty optimized for SEO so eventually it can become in first page at Google.
  • It should be easy to find this docs for newbies.
  • It probably make sense to link https://github.com/golang-standards/project-layout in this docs and explain that it's alternative point of view and have nothing with "official" Go team.
  • We should not forget about this comment
  • It make sense to add all existing references to this guide.
  • Maybe it make sense to add some other references to some articles published on internet.

Please vote where it will be better to create such guide in you opinion:

@gudvinr
Copy link

@gudvinr gudvinr commented May 2, 2021

This kind of recommendations are useless in general. Project structure only makes sense if it is enforced by language/framework tooling.
You can't force all the developers to follow whatever one thinks is "perfect layout"™ because this is just a matter of personal preference.

Only thing that is enforced by go tools is internal directory and that's all you have. Other aspects shouldn't be standardized by language since you can't say that every project is the same.

You can't use same layout for projects of different sizes.
If you split small package into some kind of "one layout for all" you'll probably end up with many packages with couple of .go files in them.
"Good" examples of this are Django, RoR and numerous JS one-click CLI project builders. Even for simple projects it creates pile of templates that you might end up never using.

Different teams also have different views on what is working for them and it may not be same for other folks.

@BinGuoGuo
Copy link

@BinGuoGuo BinGuoGuo commented May 8, 2021

Go novices said: The layout of the go project and the use of mods are really confusing, and the official knowledge is too scattered. :(

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Proposals
Incoming
Linked pull requests

Successfully merging a pull request may close this issue.

None yet