Separating "participants" from "testers"

[shankari]

In the CEO mini-pilot, we had a single instance for all users, and distinguished between participants and testers by using a new field called "install_group" that I added. Clearly, we need to only look at participant data for any future analysis.

However, as we move to the full pilot, we decided to try out a micro-services architecture in which we create separate containers for each program and for the testers. In that case, we don't need an install_group any more, don't need to filter by "participant" and can simply work with all users.

As long as the program containers are invoked the same way as the test containers, we should not need end-to-end testing against the program containers.

@asiripanich @PatGendre @jf87 @ericafenyo @kafitz do you think that future deployments will go towards a standard ops architecture with separate staging and production environments? Or do you think people will always want to test the production environment end to end and we should build in first class support for install groups?

Hint: I think that the first option is the right one. But want to check before committing to it.

[PatGendre]

@shankari that's great that you open up this issue because we are currently asking ourselves the same question :-)

We think a staging env would be a good idea, too, if we need to enhance the server side.
It is possible to work on a local server for instance on the pipelinewith test data, but for other purposes a staging server seems a must.
On the client side, the testers use a test version (closed)app the with a before we publish it.

[ericafenyo]

@shankari We also end up with staging and production instances and I think it's a good idea.

[kafitz]

Hi all! I'm also for the first approach. As I'm often tasked with setting up and managing travel survey instances, it's easiest to manage when I can safely guarantee the environments are consistent from my local testing to what's deployed.

My preferred way is using CI to build containers, test that container once and this container can then be pulled by all my environments. For the case of a feature available to testers, this could be controlled by an environment variable.

This lets me have various docker-compose.yml files for each stack which can be version controlled. It's also what I found worked best for me over the years with Itinerum since our client organizations would often want their own test instance before going live.

[shankari]

Great to hear consensus around the right approach.

I was just worried about whether the deployers/clients would want end-to-end testing against production, or whether they would trust me that the production and staging containers were deployed from the same codebase. I have not had any push back so far (first production deployment this week after ~ 1 month of running the test environment). Great to hear from @kafitz that that is likely to hold.

This is the diagram I have been showing to the deployers all along.

I could of course have different UIs for the different clients as well, but we (NREL) is doing all development, and I don't like making UI changes, so I tried to go with standardization and only resort to different channels if the requests were significantly different. So far so good 🤞

[shankari]

I also really need to improve the tooling around maintaining multiple containers at the same time. Manually running docker-compose is unlikely to cut it when I have 7 sets of containers running at the same time.