Inference performance files

[Aniket-Parlikar]

Added new files for measuring inference performance

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[ivanzvonkov]

Comments about code
Overall code looks okay. I see lots of comments which don't need to be there and repeated code that can be moved into a single file.

[ivanzvonkov]

The suggested implementation in #201 states:

• Individual function for each task which outputs performance indicators
• General script for all tasks which outputs a log/txt file with all performance indicators

1. It appears to me that some of the performance indicators are still missing, I've made a Google Sheet to track which have been recorded here: https://docs.google.com/spreadsheets/d/1_ZqWCInh8xBGglFrd4r_L2urMZG5f_U6zBdr3wy54Jk/edit?usp=sharing
   Is this accurate?

2. There is no general script which outputs a log/txt file with all performance indicators. Why not?

[ivanzvonkov]

@Aniket-Parlikar I see several comments are not yet addressed, please let me know when this is ready for a second look

[Aniket-Parlikar]

In regards to this comment,
PFA the answers below.

#213 (comment)

1. It appears to me that some of the performance indicators are still missing, I've made a Google Sheet to track which have been recorded here: https://docs.google.com/spreadsheets/d/1_ZqWCInh8xBGglFrd4r_L2urMZG5f_U6zBdr3wy54Jk/edit?usp=sharing
   Is this accurate?
   Ans: I had already uploaded files which contains the information indicated in the missing fields.Malawi_2020_September.csv(I'll rename it for better understanding) contains information regarding the performance parameters of single model inside a Docker container.

Whereas, multi_models_logs.csv contains information about the performance parameters of multiple models deployed in a Docker container.

The cloudrun_logs.txt contains information about the performance parameters of multiple models deployed on Google cloud run service.

2. There is no general script which outputs a log/txt file with all performance indicators. Why not?
   The main reason I believe is due to the fact that we intend to obtain the performance parameters of models deployed in various environments and hence, they need to run on each environment seperately. In addition, some of these parameters vary from environment to environment and we need different measurement approaches in such situations.

[ivanzvonkov]

Should this be updated based on previous comment?

Yes, there are 10 active containers for 10 requests

[Aniket-Parlikar]

Yes, these can be discarded

[ivanzvonkov]

Nice summary!!

[ivanzvonkov]

Just wondering why is it 0.5 here and 1 above?

[Aniket-Parlikar]

It was generally set as the execution time of test_models.py is pretty less as compared to concurrent_inference.py due to which while profiling the program, we can try to get performance parameters more accurately.

[ivanzvonkov]

Thanks for the updates, this is a good comprehensive overview!
I think the notebook may need to be updated to not have the duplicate result.
I left a couple small comments, once they are addressed this is ready to merge!

[ivanzvonkov]

This is request latency is probably the biggest issue! I wonder why it is growing so fast