Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

building | testing go parallelly. #39463

Closed
hongyi-zhao opened this issue Jun 8, 2020 · 10 comments
Closed

building | testing go parallelly. #39463

hongyi-zhao opened this issue Jun 8, 2020 · 10 comments

Comments

@hongyi-zhao
Copy link

@hongyi-zhao hongyi-zhao commented Jun 8, 2020

Hi,

When I compile go from source code with the following two methods:

$ git clone https://github.com/golang/go.git go.git
$ cd go.git/src
# building and testing:
$ ./all.bash
# or building only:
$ ./make.bash

Is it possible for me to do the job parallelly, i.e., like the manner of make -j N?

Regards,
HY

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Jun 8, 2020

It is already done in parallel, as controlled by the GOMAXPROCS environment variable, q.v.

(If you are aware of areas where it could usefully be more parallel, please let us know.)

@hongyi-zhao
Copy link
Author

@hongyi-zhao hongyi-zhao commented Jun 9, 2020

Thanks a lot for pointing this out. What's real value for this variable? Is it set automatically according to the number of physical cores on user's machine?

@davecheney
Copy link
Contributor

@davecheney davecheney commented Jun 9, 2020

I defaults to GOMAXPROCS

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Jun 9, 2020

GOMAXPROCS defaults to the number of logical CPU cores on your machine.

@hongyi-zhao
Copy link
Author

@hongyi-zhao hongyi-zhao commented Jun 9, 2020

@ianlancetaylor Do you mean it will account into the hyper threading results?

@davecheney
Copy link
Contributor

@davecheney davecheney commented Jun 9, 2020

Whatever your operating system counts as a core is what go build will use.

@hongyi-zhao
Copy link
Author

@hongyi-zhao hongyi-zhao commented Jun 9, 2020

Do you mean the following counting method:

$ cat /proc/cpuinfo | grep processor | wc -l
4

@davecheney
Copy link
Contributor

@davecheney davecheney commented Jun 9, 2020

The method differs by operating system and processor.

Stepping back for a moment, what is the problem you are having? You've mentioned something about building Go code in parallel. To the best of my understanding the Go tool builds packages in parallel by running multiple copies of the go compiler -- one per package, up to GOMAXPROCS by default or go build -p N if supplied. Inside the compiler each function is compiled in parallel, again using GOMAXPROCs threads. I'm reasonably confident that the Go tool, where possible, uses the resources the operating system makes available to it when compiling. Do you disagree? If so, please explain what you are seeing.

@hongyi-zhao
Copy link
Author

@hongyi-zhao hongyi-zhao commented Jun 9, 2020

@davecheney I compile go git master version on a dual Xeon E5-2699v4 (44 physical cores in total) machine with 256 G ram. It takes several minutes for buildings and testings. So I want to know whether this is reasonable.

@davecheney
Copy link
Contributor

@davecheney davecheney commented Jun 9, 2020

I think a few minutes is a reasonable time to execute ./all.bash. You can judge for yourself using a tool like top(1) or htop(1).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.