New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The Telepresence test suite could run Telepresence many fewer times #400

Closed
exarkun opened this Issue Jan 10, 2018 · 3 comments

Comments

2 participants
@exarkun
Contributor

exarkun commented Jan 10, 2018

The Telepresence test suite has approximately 30 tests and it is run around three times per platform (once for each of inject-tcp, vpn-tcp, and container). Most tests involve running the telepresence tool which must manipulate a Deployment and get a pod running the proxy container to the Running state.

Measurements suggest that this setup may take around 8 seconds. If we can convince some number of the tests to share one telepresence invocation then we may be able to shave a significant amount of time from the test suite runtime.

@exarkun

This comment has been minimized.

Contributor

exarkun commented Jan 10, 2018

One pattern we've identified in the test suite is this:

    def test_existingdeployment(self):
        """                                                                                                                                                                                                         
        Tests of communicating with existing Deployment.                                                                                                                                                            
        """
        self.existingdeployment(None, "tocluster.py")

    def test_environmentvariables(self):
        """                                                                                                                                                                                                         
        Local processes get access to env variables directly set and set via                                                                                                                                        
        envFrom.                                                                                                                                                                                                    
        """
        self.existingdeployment(None, "envvariables.py")

The existingdeployment method runs telepresence --deployment ... in such a way as to run the Python program identified by the second argument passed to it. The application-specific logic for asserting some desired behavior is implemented in the script itself (either tocluster.py or envvariables.py in the above code).

One improvement would be to merge the logic in tocluster.py and envvariables.py. This reduces the number of telepresence invocations by one, thus saving some overhead.

This idea can hopefully be generalized, merging most or all of the necessary logic which must run in the Telepresence "context" (where tocluster.py and envvariables.py above are being run) into a single program. This can be run with one invocation of telepresence. This changes the overhead from O(N) on the number of tests to O(1).

To aid in developing further tests and debugging failures, rather than simply and straight-forwardly combining all of these programs into one, we want to instead to change the factoring somewhat.

Instead of taking actions and checking the validity of the results, we want the program which runs in the Telepresence context to gather data and expose it to the pytest process. The data can be gathered up and then test methods can inspect the data and make assertions about it. This allows us to maintain a useful separation of the testing logic into different test methods while accomplishing the goal of reducing the per-test overhead.

@ark3

This comment has been minimized.

Contributor

ark3 commented Mar 9, 2018

Reevaluate #198 when closing this.

ETA: #198 is not addressed yet, but should be straightforward now.

@ark3

This comment has been minimized.

Contributor

ark3 commented Mar 14, 2018

Add "how do tests work?" documentation before closing this.

ETA: #509 does this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment