Assignment 3 final

Final Problem Statement

Contributing different features to the Apache OpenSource project Airavata-MFT. We try to focus on the improvements with respect to adding new transport, building different clients for the Airavata-MFT. Also, as part of assignment 3, we are focusing on increasing the test coverage for all the transports protocols and follow test-driven development, an industry-standard, for the protocol that we are developing. We are now concentrating on the different transport modules in the project.

Differences from Initial Problem Statement

The initial problem statement focused on building service mesh on top of Project 2 that we had built. We decided to change the approach during phase 1 as we reevaluated our course preference and we wanted to contribute to the Apache-MFT project. Also, we initially thought that meaningful contribution to Airavata-MFT would be difficult given the time constraint.

Problem Statement Development

We initiated several conversations on the developer mailing list. In fact, when team Pika Pika developed the new transport mechanism and we were following the conversations actively in the mailing list. Below are the links to the conversations where we came up with a problem and initiated a solution for it:

New Transport Support: (FTP)

https://www.mail-archive.com/dev@airavata.apache.org/msg12645.html

New Client Support (Python) https://www.mail-archive.com/custos@airavata.apache.org/msg00135.html https://www.mail-archive.com/dev@airavata.apache.org/msg12721.html

Improving the code coverage

https://www.mail-archive.com/dev@airavata.apache.org/msg12720.html

Methodology

We started by going through and understanding the architecture of the Apache Airavata MFT. We forked the repo from the Airavata-MFT and played around did some testing in the local. After resolving initial dependency issues, we were able to successfully run the code in our local systems. From our investigation, we saw the opportunity to develop support for new transport. However, we could contribute to only one transport although we wanted to contribute more improvements on transport. As far as developing additional client is concerned we were inspired by one of the other Airavata projects (Airavata Custos) where there are multiple language clients. We wanted to implement that in Airavata-MFT. As far as the code coverage is concerned, this was right in front of our noses. Except mft-core module, none of the other modules had any test cases. The tests written in the mft-core module are not even real tests. Hence we took the initiative to write unit tests following the industry standards and added almost 100% coverage to all the classes we added test cases to.

Implementation:

Firstly we brainstormed on the MFT architecture. There were implementations of SCP, Local, S3 already implemented. With a bit of code analysis, we came to know what needs to be done. Since the code is already modularized, understanding how to implement new transport was very clear.

contribution - https://github.com/apache/airavata-mft/commits?author=gkiran292

For client implementation, we took inspiration from Apache Airavata Custos. However, implementation was a bit different as we wanted to generate the python code from stubs from the POM plugin. It is evident for java as it has maven support, integrating proto plugin for python plugin got us to do some serious digging. Although protobuf-maven-plugin mentions the way to integrate the plugin, it was not obvious. We had to build the plugin from the source to integrate that plugin in the project. Well, we can now integrate the other language plugins with protobuf-maven-plugin by building the proto plugin for their respective languages from the source.

contribution - https://github.com/apache/airavata-mft/pull/17/commits

For improving the code coverage we stuck to the basics where we first tried it in our local by refactoring few necessary classes and went ahead with the unit test cases as there were none already.

contribution - https://github.com/apache/airavata-mft/pull/18/commits

One can look at the sheer volume of tests that were written to cover all the edge cases for the various transport protocols in MFT. In total, 207 test cases were added with the following split: Tests: TestConnectorResolver.java for ConnectorResolver.java - 25 test cases TestFileBasedResourceBackend.java for FileBasedResourceBackend.java - 16 test cases TestFileBasedSecretBackend.java for FileBasedSecretBackend.java - 12 test cases TestMetadataCollectorResolver.java for MetadataCollectorResolver.java - 9 test cases TestResourceServiceHandler.java for ResourceServiceHandler.java - 80 test cases TestSecretServiceHandler.java for SecretServiceHandler.java - 52 test cases TestSQLResourceBackend.java for SQLResourceBackend.java - 13 test cases

In total, 4190 lines of code were added in this commit.

These test cases cover all edge cases making a developer life simple during his/her next contribution.

Evaluation

Identify the issues: The problems we could tackle was either we could expand on the service we has already created. This would add abilities like dynamic load balancing and scaling of the system to adapt, system security, etc. The other issue we faced was the technical difficulty of understanding the entire architecture of MFT or Cust OS, then see what improvements were needed in them.
Understand everyone's interests: The three team members each wanted to do open source contributions but were not sure about the possibility of doing such in the limited time.
List the possible solutions (options): WE prepared for doing the service mesh while individually trying to understand the MFT and custos codebases. Then we brainstormed together for possible contributions we could make.
Evaluate the options: We found these contributions possible

Implement a service mesh. for MFT:
New Transport Support: (FTP)
New Client Support (Python)
Improving code coverage

Select an option or options: Each of the team members picked up tasks most suited to their skill sets and experience levels.
Document: We updated wikis for each step of the process.
Agree on contingencies, monitoring, and evaluation: We reached out to regular contributors and maintainers of the codebases about our ideas, implemented them, and asked for improvements. We went through some good refactoring and enhancements before these were accepted.

Conclusions and Outcomes

The MFT codebase is very well structured. Each and every module is so intuitive and well modularized. We especially appreciate how the stub information is passed on to different modules via the dependency. In fact, in our previous projects, we used to transport the proto files every time and it didn't occur to us to build it and transport it as a library. This may be very small and subtle, but we are definitely taking forward this learning for the future. This also helped a bit in our knowledge about version control and Git. Creating relevant pull requests and having code reviewed by others definitely brings the desired outcomes of our problem statements.