Testing?

[philippefutureboy]

Is your feature request related to a problem? Please describe.
I'm currently setting up a TDD flow for cubejs. I have two questions:

1. What would one test with regards to cubejs? So far on my side I think that testing the schemas for proper output seems to be a good rule of thumb.
2. Is there already a mechanism integrated to cubejs for testing?

Describe the solution you'd like
Cubejs provides an npm command that allows to run test against a mock database. Doesn't require extra setup bar installing dependencies (and potentially having docker/docker-compose) installed.

Describe alternatives you've considered
I've implemented a docker-compose setup with jest for integration testing (e2e). It runs cube in its own docker environment and uses jest to fill the database, issue queries using @cubejs-frontend/core, and test the result. Now I was wondering if there would be a way for such flow to be either done without a database, or done similarly, but without the need for extra setup like I did.

Additional context
If you are interested I can share my test setup repository :)

[philippefutureboy]

Heyo!! I've added my test setup repository! 🎉

https://github.com/arthurintelligence/cubejs-test

[paveltiunov]

@philippefutureboy Hey Philippe! Thanks for sharing!

That's really great question. So with schema compiler we use exactly same approach: integration tests which fill the database, run queries and test query results. Hard part here is you always need a database because cube.js schema most of times is DB specific. There was discussion about driver test suite while clickhouse implementation. I think it should be a separate package for testing cube.js which sets up environment like docker, loads the fixture to db and executes queries.

[philippefutureboy]

@paveltiunov Hey Pavel! Thanks for your answer :)

Hard part here is you always need a database because cube.js schema most of times is DB specific.

Agreed, but we can generate docker-compose configurations dynamically and then create the containers on the fly.

There was discussion about driver test suite while clickhouse implementation.

How would clickhouse help with SQL dialect specificity?

I think it should be a separate package for testing cube.js which sets up environment like docker, loads the fixture to db and executes queries.

Agreed, I think that would be the optimal workflow. A package which reads your current configuration and can act as the test agent for your codebase. It would dynamically manage your docker environments based on a configuration file or by executing the index.js with mocked drivers to determine the configurations (I'd opt for a config file, easier to implement and maintain).

Do you think my repo could be a good basis to start building that package? You most likely have something more thorough internally - would be great to see what you have in store

🚀

[paveltiunov]

@philippefutureboy I mean it was discussed here a little bit: #98.

Yeah. We was thinking about it internally and here's the list of requirements for such test package we came up with:

1. Should be test framework agnostic.
2. Should provide simple API to init database, fill it with provided fixture, run query and then tear down it.
3. Ideally server-core shouldn't be involved here otherwise it complicates testing. Test package should use driver, schema-compiler and optionally api-gateway to create concurrency free testing environment.

[philippefutureboy]

1. How do you intend to make it test-framework agnostic?
2. By simple API to init db, you mean that each test suite would be able to create a db within the database instance?
3. How does server-core complicate things when involved?

[paveltiunov]

@philippefutureboy
I was thinking about following interface:

const cubejsTest = new CubejsTest({
  prepareFixture: (driver) => {}, // use driver to prepare fixture
  // various common params for CubejsServerCore goes here like driverFactory if required
  // based on a driver provided appropriate DB container should be started
});

describe('Visitors', function test() {
  this.timeout(20000);

  after(async () => {
    await cubejsTest.tearDown();
  });

  it('should calculate revenue', async () => {
    // here's the case with lazy init
    // on first runQuery call container should be started, filled with fixture and then query should be executed
    const result = await cubejsTest.runQuery({
      measures: [
        'visitors.visitor_revenue'
      ],
      dimensions: ['visitor_checkins.cardsCount'],
      timezone: 'America/Los_Angeles',
      order: [{
        id: 'visitor_checkins.cardsCount'
      }]
    });
      
    result.should.be.deepEqual([
      {
        "visitor_checkins.cards_count": "0",
        "visitors.visitor_revenue": "300"
      },
      {
        "visitor_checkins.cards_count": "1",
        "visitors.visitor_revenue": "100"
      },
      {
        "visitor_checkins.cards_count": null,
        "visitors.visitor_revenue": null
      }
    ]);
  });
});

So answering to your questions:

1. Please refer to example above. Even if we use test framework there's no actual depend from CubejsTest itself.
2. No as most of tests are read-only. For those ones which require pre-aggregations temp tables should be used behind the scenes.
3. server-core uses query-orchestrator which is caching, renewing data and maintains query execution queue. All of that can create various race conditions during the tests.

[philippefutureboy]

So if I understand properly, what you propose is to have a library CubejsTest that receives the same drivers and options as CubejsServerCore, but that is also responsible for spinning their own container for the database.
While you say that most tests are read-only, there's still a need to setup the data in the tables. Different tests may require different datasets, and you wouldn't want two tests to collide. Plus what happens when you teardown in one test? Does it wait for the other tests to resolve before tearing down?

Hence I think that for every CubejsTest instance, a new container would be spun up.
However does that scale? If not, is there an optimization that can be done by creating only one container for each type of driver employed and then simply address different database names/documents?

[paveltiunov]

@philippefutureboy

So if I understand properly, what you propose is to have a library CubejsTest that receives the same drivers and options as CubejsServerCore, but that is also responsible for spinning their own container for the database.

Yep. Right.

While you say that most tests are read-only, there's still a need to setup the data in the tables. Different tests may require different datasets, and you wouldn't want two tests to collide. Plus what happens when you teardown in one test? Does it wait for the other tests to resolve before tearing down?

That's a great question. I guess we need to manage containers inside CubejsTest instance and not statically. This way you can use different environments for different tests without collisions on one hand and re-use environment when we need to on other hand. Usually tests are run by all test frameworks consequently. If there's some sort of concurrency there it can be solved using teardown mutex or something like this.

Hence I think that for every CubejsTest instance, a new container would be spun up.

Yes. Right.

However does that scale? If not, is there an optimization that can be done by creating only one container for each type of driver employed and then simply address different database names/documents?

You can re-use single CubejsTest for all tests within a project or at least for a test suite. We have already tests that work with this level granularity and it seems to be working solution.

We would love if you can contribute such test package.

Thanks!

[philippefutureboy]

@paveltiunov I'd be happy to do so! How can I be of help?

Also, I attended a JS meetup Tuesday, and turns out the subject was about testing JS/React/GraphQL using Jest/Enzyme. Given the similarities between your @cubejs-client/react interface and the one provided by GraphQL, I think Cubejs React would benefit to emulate the GraphQL structure when it comes to testing => https://www.apollographql.com/docs/react/recipes/testing/#mockedprovider

What do you think?

Have a great day ✨

Phil

[paveltiunov]

@philippefutureboy I mean if you got the idea you can contribute such package to cube.js repository. We'll release it under @cubejs-backend then.

Also, I attended a JS meetup Tuesday, and turns out the subject was about testing JS/React/GraphQL using Jest/Enzyme. Given the similarities between your @cubejs-client/react interface and the one provided by GraphQL, I think Cubejs React would benefit to emulate the GraphQL structure when it comes to testing.

I think it's neat idea. We're still figuring out what should be tested on frontend side as well so it worth considering.

Thanks!

[philippefutureboy]

Great! Will get started sometime next week :)

[philippefutureboy]

@paveltiunov I'm going through the cubejs-backend/* codebase right now, and I can't seem to find a test suite for the code. Do you have any tests & CI/CD pipeline?

[paveltiunov]

@philippefutureboy Most of tests are in cubejs-schema-compiler package. We don't have CI but we test before each release. Need to setup one for sure.

[philippefutureboy]

@paveltiunov I think that should be a major priority – your clients as well as those using cube.js as part of their project most likely use cube in production deployments. What happens if in a new patch there is a bug that is inserted? How do you know about regressions?

I'd volunteer to add a few tests here and there, but I don't know the internals of cube.js. If you are ready to provide me with BDD-style tests ([given x,] should do y) and descriptions of the expected behaviour, I can try my hand at helping you raise your test coverage during my spare time.
It would take priority over @cubejs-backend/server-test package however.

[paveltiunov]

@philippefutureboy That's really great point. The thing is cube.js right now has 2 components with decent amount of logic. These are schema-compiler and query-orchestrator. First one is heavily tested and at least 90% test covered. The second one is heavily tested in production environments and since it was open sourced there were no major changes to core components. There's 30-40% test coverage for query-orchestrator. And other backend components are merely wiring code.

We definitely need to increase code coverage. And I think we should start adding tests as new code changes come in or new regressions are spotted. Given current code base is tested by production deployments which manage over a dozen trillions of data points in total this lazy test adding approach seems something we can afford right now.