Assignment 3 final

Final Problem Statement

Contributing different features to the Apache OpenSource project Airavata-MFT. We try to focus on the improvements with respect to adding new transport, building different clients for the Airavata-MFT. Also, as part of assignment 3, we are focusing on increasing the test coverage for all the transports protocols and follow test-driven development, an industry-standard, for the protocol that we are developing. We are now concentrating on the different transport modules in the project.

Differences from Initial Problem Statement

The initial problem statement focused on building service mesh on top of Project 2 that we had built. We decided to change the approach during phase 1 as we reevaluated our course preference and we wanted to contribute to the Apache-MFT project. Also, we initially thought that meaningful contribution to Airavata-MFT would be difficult given the time constraint.

Problem Statement Development

We initiated several conversations on the developer mailing list. In fact, when team Pika Pika developed the new transport mechanism and we were following the conversations actively in the mailing list. Below are the links to the conversations where we came up with a problem and initiated a solution for it:

New Transport Support: (FTP)

https://www.mail-archive.com/dev@airavata.apache.org/msg12645.html

New Client Support (Python) https://www.mail-archive.com/custos@airavata.apache.org/msg00135.html https://www.mail-archive.com/dev@airavata.apache.org/msg12721.html

Improving the code coverage

https://www.mail-archive.com/dev@airavata.apache.org/msg12720.html

Methodology

We started by going through and understanding the architecture of the Apache Airavata MFT. We forked the repo from the Airavata-MFT and played around did some testing in the local. After resolving initial dependency issues, we were able to successfully run the code in our local systems. From our investigation, we saw the opportunity to develop support for new transport. However, we could contribute to only one transport although we wanted to contribute more improvements on transport. As far as developing additional client is concerned we were inspired by one of the other Airavata projects (Airavata Custos) where there are multiple language clients. We wanted to implement that in Airavata-MFT. As far as the code coverage is concerned, this was right in front of our noses. Except mft-core module, none of the other modules had any test cases. The tests written in the mft-core module are not even real tests. Hence we took the initiative to write unit tests following the industry standards and added almost 100% coverage to all the classes we added test cases to.

Implementation:

Firstly we did a brainstrom of the MFT architecture. There were implementations of SCP, Local, S3 already. With a bit of code analysis we came to know what needs to be done. The code so modularized that understanding how to implement new transport was as clear as day.

contribution - https://github.com/apache/airavata-mft/commits?author=gkiran292

For client implementation we took inspiration from Apache Airavata Custos. However, implentation was a bit different as we wanted to generate the python code from stubs from the POM plugin. It is evident for java as it has maven support, integrating proto plugin for python plugin got us to do some serious digging. Although protobuf-maven-plugin mentions the way to integrate the plugin, it was not obvious. We had to build the plugin from the source to integrate that plugin in the project. Well, we can now integrate the other language plugins with protobuf-maven-plugin by building the proto plugin for thier respective languages from source.

contribution - https://github.com/apache/airavata-mft/pull/17/commits

For improving the code coverage we stuck to the basics where we tried with local and went ahead with the unit test cases as there had been none already

contribution - https://github.com/apache/airavata-mft/pull/18/commits

Evaluation

Identify the issues: The problems we could tackle was either we could expand on the service we has already created. This would add abilities like dynamic load balancing and scaling of the system to adapt, system security, etc. The other issue we faced was the technical difficulty of understanding the entire architecture of MFT or Cust OS, then see what improvements were needed in them.
Understand everyone's interests: The three team members each wanted to do open source contributions but were not sure about the possibility of doing such in the limited time.
List the possible solutions (options): WE prepared for doing the service mesh while individually trying to understand the MFT and custos codebase. Then we brain stormed together for possible contributions we could make.
Evaluate the options: We found these contributions possible

Implement a service mesh. for MFT:
New Transport Support: (FTP)
New Client Support (Python)
Improving code coverage

Select an option or options: Each of the team members picked up tasks most suited to their skill sets and experience levels.
Document: We wikis for each step of the process.
Agree on contingencies, monitoring, and evaluation: We reached out to regular contributors and maintainers of the code bases about our ideas, implemented them, and asked for improvements. We went through some good refactoring and enhancements before these were accepted.

Conclusions and Outcomes

The MFT codebase is very well structured. Each and every module is so intuitive and well modularized. We especially appreciate how the stub information is passed on to different modules via the dependency. In fact in our previous projects, we used to transport the proto files every time and it didn't occur to us to build it and transport it as a library. This may be very small and subtle, but we are definitely taking forward this learning for the future. This also helped a bit in our knowledge about version control and Git. Creating relevant pull requests and having code reviewed by others will it brings desired outcomes of our problem statements instilled a learning curve in the team members