Code flow/documentation

[shivramsrivastava]

It will be great if there is some documentation on the code flow.
I tried with Doxygen, but it was not generating any docs :-(
If you can give some pointer like what are the main structures, and where to start it will great.

Is there any reason you chose c++, which other languages like golang or python can't do?
Do you have any plans on porting to other languages, if all the said features are achievable ?

[ms705]

Hi @shivramsrivastava,

Thanks for asking! You're quite right that more documentation would be helpful :-) As a small research project primarily developed by two people with only 24 hours in the day, we've never had time to write extensive documentation. However, we're trying to, in general, write well-commented code, especially in header files defining key interfaces (e.g., cost models, schedulers, executors). Some of the existing comments could probably easily be turned into Doxygen comments.

At a higher level, our work is also described in a bunch of research publications (e.g., our upcoming OSDI paper and my PhD thesis), but these have limited detail on implementation-level structure.

As for the choice of C++ as a language, it comes down to three reasons:

1. Some of our code on the critical path to scheduling decisions must be quite efficient: for example, traversing and updating the large flow graph in the flow scheduler is substantially faster in C/C++ than in a managed, garbage-collected language.
2. When we started this project, Go wasn't as mature as it is now and didn't have as large a community.
3. We're very familiar with C++, and there are many libraries available that easily link to C/C++ codebases.

That said, CoreOS in ksched have reimplemented some of our code in Go (mostly scheduler code), which might be worth a look. However, ksched is a Kubernetes scheduler plugin only and does not contain the cluster manager or simulator components of Firmament. It also does not currently implement the more complex scheduling policies (cost models) that we have implemented, and its scheduling throughput is somewhat lower than that of the C++ code.