Separate emulator ports for component and E2E tests

[Derek-Hardy]

• Environment: objectify-v6-migration branch

Currently both BaseE2ETestCase and BaseComponentTestCase extends from BaseTestCaseWithObjectifyAccess and therefore share the same EMULATOR_PORT.

It would be better to use separate ports for two kinds of tests that require local datastore emulator setup. To avoid potential interference in persisting storage.

[wkurniawan07]

E2E test should in fact not rely on automatic setup of emulator.

• It's a black-box test; the software to be tested is supposed to be ready beforehand. By having this setup, we're mixing up the step of preparing the software with the tests themselves.
• At the current setup:
  • The local datastore helper will trigger even when running test against staging server, which is a waste of resource.
  • The component tests will always use port 8484 and will possibly interfere with development.
  • Take a look at the failed L&P tests, and guess what is the missing step that causes them to fail.

If the E2E tests in CI require datastore emulator, then we should set up the emulator beforehand in the CI script.

[Derek-Hardy]

@wkurniawan07 My thoughts:

E2E test should in fact not rely on automatic setup of emulator

Yup, then we empty-override the local datastore helper setup for e2e.

we should set up the emulator beforehand in the CI script

I looked at it. setup-gcloud can be the way to do it. But I can't find a way to interpolate variables from build.properties. We might need to hardcode port number for gcloud command in e2e.yml file?

The component tests will always use port 8484

I am thinking of defining app.localdatastore.component.port for component tests and app.localdatastore.e2e.port for e2e tests in build.template.properties with different default port number

guess what is the missing step that causes them to fail

The problem could be from persistTestData() which makes backdoor API call. However, our Objectify test setup points to local datastore emulator which is non-exist in LNP tests.

So the solution should be adding its setup to test suites (i.e. extends BaseLNPTestCase from BaseTestCaseWithDatastoreAccess).

[wkurniawan07]

We might need to hardcode port number for gcloud command in e2e.yml file?

No issue with this. Tests in CI work by pulling values already hard-coded in the version control.

setup-gcloud can be the way to do it.

Might be, but you might also want to check first if using the more direct method of just running gcloud beta emulators datastore start will work.

app.localdatastore.component.port ... app.localdatastore.e2e.port ... in build.template.properties

build.properties has no business with any kind of tests.

So the solution should be adding its setup to test suites

This is right if L&P tests are white-box, however they are black-box tests and thus the same applies.

[wkurniawan07]

Work is done