Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improved task scheduling for parallel and distributed compilation based on memory/cpu usage #2571

Closed
3 of 4 tasks
waruqi opened this issue Jul 15, 2022 · 3 comments
Closed
3 of 4 tasks

Comments

@waruqi
Copy link
Member

waruqi commented Jul 15, 2022

Is your feature request related to a problem? Please describe.

  • parallel compilation
  • distributed compilation
  • get memory usage info
  • get cpu usage info

With limited memory resources inside docker, ensure that the project can be compiled successfully.

Describe the solution you'd like

...

Describe alternatives you've considered

No response

Additional context

No response

@waruqi waruqi added this to the v2.7.1 milestone Jul 17, 2022
@waruqi
Copy link
Member Author

waruqi commented Jul 28, 2022

get meminfo

$ xmake l os.meminfo
{
  availsize = 4755, -- MB
  totalsize = 16381, -- MB
  usagerate = 0.7097246810329, -- 70%
  pagesize = 4096 -- bytes
}

@waruqi waruqi changed the title Improved task scheduling for parallel and distributed compilation based on memory state Improved task scheduling for parallel and distributed compilation based on memory/cpu usage Jul 28, 2022
@waruqi waruqi mentioned this issue Jul 28, 2022
@waruqi
Copy link
Member Author

waruqi commented Jul 28, 2022

get cpu info

$ xmake l os.cpuinfo
{
  model = 142,
  family = 6,
  model_name = "Intel(R) Core(TM) i7-8569U CPU @ 2.80GHz",
  usagerate = 0.31167268753052,
  features = "fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clfsh ds a
cpi mmx fxsr sse sse2 ss htt tm pbe sse3 pclmulqdq dtes64 mon dscpl vmx est tm2 ssse3 fma cx16 t
pr pdcm sse4_1 sse4_2 x2apic movbe popcnt aes pcid xsave osxsave seglim64 tsctmr avx1_0 rdrand f
16c",
  march = "Kaby Lake",
  ncpu = 8,
  vendor = "GenuineIntel"
}

@waruqi waruqi mentioned this issue Jul 28, 2022
@waruqi
Copy link
Member Author

waruqi commented Jul 30, 2022

parallel compilation

I don't know how to control it based on memory usage. Because the memory usage is very low before multiple parallel tasks are started.

@waruqi waruqi closed this as completed Jul 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant