-
Notifications
You must be signed in to change notification settings - Fork 89
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
some questions of your code #30
Comments
If I'm not mistaken, we developed the project with tensorflow version |
thank you! |
hello, |
@chenjw259 Based on your last question, did you have an answer about it> can you provide some details if you are able to understand it. |
Re DAGs input to the model: we pass in all DAGs as input. |
I'm kind of confused of one part of Decima. You mentioned " We evaluate Decima in Simulation and in a real Spark Cluster".
|
This repo is only the simulation part of the project. We did the training purely on simulation. The real Spark integration is through a customized extension. We also modified the scheduling module of Spark to request scheduling decision from our customized extension (in python) via protobuf. This part of the code was a bit intricate and we didn't have enough time to properly refactor for public use. Will give an update if we find time to go back and open source the code. Thanks! |
Thank you so much for your response. If you don't mind, can you please give me some instructions like which part of the Spark Scheduling model code you modify it (as Spark code written in Scala and I tried it did not work it gives me an error when I did Spark-submit) and How to customize that extension to integrate with Spark using Decima code? Is it like You changed the Spark codebase or just modified some part of that Scheduling module to let your model execute in that customized extension. |
I was reading some stuff about Protobuf: https://developers.google.com/protocol-buffers ; But I am kind of confused with the modified scheduler in spark and then request scheduling decision from Your Customized extension via protobuf. How does this process actually can be done as Decima code base is huge, it will be nice if you share some details about this integration process. |
Hello! I would like to ask what are the versions of the various libraries you use for this code?
Thank you!
The text was updated successfully, but these errors were encountered: