Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Question: how to unit test without hitting the DB #1267

Open
adrianhara opened this issue Nov 29, 2017 · 60 comments
Open

Question: how to unit test without hitting the DB #1267

adrianhara opened this issue Nov 29, 2017 · 60 comments

Comments

@adrianhara
Copy link

I'm trying to write some unit tests of code that uses typeorm without hitting the DB. I'm using getRepository in the code to do stuff.

In my tests ideally I'd like to call getRepository(SomeEntity) and then use sinon to sinon.mock(repository).expects('find').withArgs(...).returns(stuff). To avoid hitting the DB I thought I'd just not call createConnection if I'm in the unit tests context.

However I ran into the problem that getRepository calls connection manager which looks if it has a connection initialized and if not throws. What would be the recommended way of handling this? Is it possible to create a "dummy" connection somehow which doesn't try to connect anywhere?

@mscharley
Copy link

Have a look over here: #171 (comment)

@adrianhara
Copy link
Author

Thanks, I had seen that one. However I'm not using testdouble.

The current (working) setup i have is this:

import * as typeorm from "typeorm";

function createFakeRepository() {
    return {
        find: function() {},
        findOne: function() {}
        // others
    }
}

// then in a test
 const fakeRepository = createFakeRepository();
 sinon.stub(typeorm, 'getRepository').withArgs(Profile).returns(fakeRepository);

 const profileRepositoryMock = sinon.mock(fakeRepository);
 profileRepositoryMock.expects('find').withArgs({emailAddress: profile.emailAddress}).returns(promise.resolve([profile]));

It works but is not pretty at all.

@mscharley
Copy link

I'm not using testdouble.

You easily could be, though. We use both testdouble and sinon, they're both good at different things. testdouble is very good for this particular problem. sinon, not so much.

@pleerock
Copy link
Member

can you guys elaborate more on this question? Testing strategies aren't defined by typeorm yet and I would like to make them and provide all necessary tools for this purpose. Would be great if each of you describe about what testing-helper tools they would like to have in typeorm and provide a real world examples how they can be used.

@mscharley
Copy link

For unit tests, using the approach I mentioned in the other thread you don't need anything from typeorm, it just works. That's why I suggested it. With minimal code in your tests It works today and works great.

For integration tests you probably do actually want a real database of some sort. Either a separate database/schema in your existing local dev database or a temporary sqlite database. If typeorm doesn't already then supporting some form of purely in-memory database may be a good thing here as well.

Expanding on my last comment, testdouble does mocking really well. sinon struggles in lots of ways. In particular, testdoubles ability to use ES2015 Proxy objects as mocks is really awesome. but testdouble only does mocking, there's lots of other utilities in sinon that are really awesome. it's just the mocking tools that are kind of lack-lustre.

@ProofOfKeags
Copy link

Yeah I actually would prefer an in memory db implementation for testing to increase the speed of my automated tests and to decrease the need for a db to actually need to run for the sake of interface testing. Obviously there comes a stage where it would be nice to test out the full db but for automated regression testing it slows you down by an order of magnitude. What would it take to actually add this functionality for a ':memory:' driver?

@pleerock
Copy link
Member

pleerock commented Dec 4, 2017

As I already mentioned we should start with a detailed proposal, suggestions and real-world scenarios

@mscharley
Copy link

I'll just repeat my previous comments with a little extra justifications and leave it at that. TL;DR version: typeorm doesn't need anything new implemented, in my opinion.

There's two cases I see here: unit tests and wider tests (integration, system, whatever you want to call them). Unit tests are defined as testing individual units in isolation, most commonly single functions or classes in isolation.

For unit testing there are mocking frameworks that can mock out the database entirely negating any need to have any sort of typeorm anything. Between the two tickets open about this there are examples in both the testdouble library and Sinon. The Sinon example is more verbose but hardly difficult or particularly long. Specifically, you don't want an in-memory database because you don't want to rely on (or ideally, not even load) typeorm in your unit tests. You simply want something that looks like typeorm - a mock.

For your wider system testing you want the same database as your production system will use. For wider tests you're sacrificing speed for correctness. Lets take an example from a project I'm currently working on. We use Postgres in production. We use a lot of Postgres specific stuff, specifically we use the uuid column type everywhere and also make pretty heavy use of JSONB. Should we expect those features to be supported in a custom in-memory database? Even if you don't use any custom stuff, SQL is pretty loosely specified. MySQL uses backticks for quoting table names. T-SQL uses square brackets. Postgres uses normal quotes. How do you resolve issues about basic SQL syntax? And at the end of the day, even if you solved all those issues, many of the database engines already have this solved anyway - MySQL provides the MEMORY storage engine for instance. And if you are using something that hasn't then you can always create a RAM disk for the database's data files while testing if they aren't already performant enough for your needs.

@icepeng
Copy link

icepeng commented Dec 11, 2017

IMO, it would be most effective to create an example of how to mock the typeorm repository with testing frameworks. It will be also good to utilize typedi together.

@hmil
Copy link
Contributor

hmil commented Dec 11, 2017

@CaptJakk That's what the sqljs driver is for.

I do agree with @mscharley for the most part but if I'm unit-testing a class using the DB I like to write assertions on the result rather than on the way the result was obtained (for instance you could easily mess up an assertion on FK or UNIQUE integrity where an actual DB provides it for free).
But then I guess it depends on context. And testdouble.js looks super nice to mock the typeorm API 👍

Example unit test with jest
describe('The Executor', () => {

    let executor: Executor;
    let connection: Connection;

    beforeEach(async () => {
        connection = await createConnection({
            type: 'sqljs',
            entities: [
                LogEntry,
                User,
                Topic
            ],
            logging: false,
            dropSchema: true, // Isolate each test case
            synchronize: true
        });
        executor = new Executor(connection);
    });

    afterEach(async () => {
        await connection.close();
    });

    describe('applyOperation', () => {
        it('inserts entities', async () => {
            const user = new User();
            user.id = 'testId';
            user.name = 'Jean test';
            const operation = insertOperation(
                user,
                'user',
                23
            );

            await executor.applyOperation(operation);
            const result = await connection.getRepository(User).findOneOrFail('testId');
            expect(result).toEqual(user);
        });
    });
});

@VinceOPS
Copy link
Contributor

VinceOPS commented Feb 3, 2018

Same issue here - creating a custom repository involves creating a valid connection.
I had a look at the code but mocking Connection, EntityManager, etc, seems to be a lot of work.

Started using sqljs but it does not support all datatypes (e.g. timestamp, which is valid with Postgres). As it's a port of SQLite, it should support all datatypes. I created an issue regarding this specific point.

Mocking everything still sounds as the best solution to me, though, it's easier said than done.

@ghost
Copy link

ghost commented Mar 10, 2018

Is it possible to have a "test mode" on Typeorm in which every operation happens inside a transaction that is never committed?

@szkumorowski
Copy link

szkumorowski commented Mar 28, 2018

@ericaprieto totally agree - this is a way how it should be solved
@pleerock

Spring framework has a really good support for testing, especially running each test in own transaction.
Transaction is runned with read uncommited isolation, then nothing fails due to lack of data.

As with above after completed test the transaction is never commited - roll back makes the database state constistent across all test cases.

I know here is coming different nature of nodejs and java - especially thread local context, however this is great solution and would be great to somehow allow developers to write tests in this way even with run tests one by one instead of concurrent execution of them.

Above solution will allow to build e2e/integration tests in predictable and simple way with possiblities to prepare data per test case or change database state during test execution.

@KyleGalvin
Copy link

I have a project that hooks into travisCI on push, and there is no database in that environment to run tests against.

I have been fussing with mock libraries on and off for weeks to try and get a stable solution.

I had a 'eureka' moment last night and discovered there is an in-memory database option via sql.js. This allows travis to run e2e tests, and alleviates my current mock nightmare. win-win.

I'm concerned I'll run into the timestamp issue documented here:
#1543

I'll be trying it out myself tonight. If my anecdote can help, or if you want code snippets to use as documentation, I'd be happy to offer assistance improving docs related to testing.

@ghost
Copy link

ghost commented Jul 17, 2018

Can anyone provide working snippets and docs for this? Thanks!

@pburrows
Copy link

I created a StackOverflow question for this:
https://stackoverflow.com/questions/51482701/how-do-you-mock-typeorms-getmanager-using-testdouble

If anyone wants to take a stab for credit (or for anyone stumbling on this in the future).

@oeddyo
Copy link

oeddyo commented Aug 10, 2018

would love to have some working examples, especially for data mapper pattern unit-testing!

@brdk
Copy link

brdk commented Nov 15, 2018

For testing this part of code

const repository = getRepository(TopProject);
const topProjects = await repository
        .createQueryBuilder('tp')
        .innerJoinAndMapOne('tp.customer', Customer, 'c', 'tp."customerId" = c.id')
        .innerJoinAndMapMany('tp.resourceTopProjects', ResourceTopProject, 'rtp', 'tp.id = rtp."topProjectId"')
        .innerJoinAndMapOne('rtp.resource', Resource, 'r', 'rtp."resourceId" = r.id')
        .where('tp."isActive" = :active', {active: true})
        .getMany();

// here some operation with result

I use sinon

import * as typeorm from 'typeorm';

const fakeQueryResult = [];

const fakeSelectQueryBuilder = sinon.createStubInstance(typeorm.SelectQueryBuilder);
fakeSelectQueryBuilder.innerJoinAndMapMany.returnsThis();
fakeSelectQueryBuilder.innerJoinAndMapOne.returnsThis();
fakeSelectQueryBuilder.where.returnsThis();
fakeSelectQueryBuilder.getMany.resolves(fakeQueryResult);

const fakeRepository = sinon.createStubInstance(typeorm.Repository);
fakeRepository.createQueryBuilder.returns(fakeSelectQueryBuilder);

const stubGetRepository = sinon.stub(typeorm, 'getRepository').returns(fakeRepository);

@RobinCK
Copy link
Contributor

RobinCK commented Jan 10, 2019

@pleerock
It would be nice to have interfaces.
For mock instance

example:

export class Connection implements IConnection {
// ...
}

and other.
Use case: I have typeorm extension

export class Builder {
    constructor(private readonly connection: Connection, /* ... */) {}
}

in order to write a unit test, I need to pass a Connection instance
but if there were interfaces I could write the code more freely and specify the interface.

export class Builder {
    constructor(private readonly connection: IConnection, /* ... */) {}
}

and create mock for me specific logic.
If you like going, I could make these changes.

@pleerock
Copy link
Member

@RobinCK no, we don't need interfaces. Since TypeScript is using structural typing, connection: Connection isn't necessary to be type of class Connection, it can also be class YourMockedConnection which can implement Connection (yes, you can implement class, not only interface) and can NOT implement it - no matter, structural typing will work anyway. Its not stupid Java dude! 😆

@joseym
Copy link

joseym commented Jan 21, 2019

After banging my head against this for a few days I ended up writing a small "utility" file that bypasses the DB connection via class overrides.

I haven't fully tested it but I've so far been able to create entities and run custom repository methods.

I plan on extending the utility to import raw JSON fixtures into memory.

I'm hoping that I dont run into too many issues with this, as it seems promising so far.
... Even better would be proper unit test utilities built into TypeORM!

Here are the files

dbUtility.ts

This file stubs out the createConnection method, and overrides core class methods to bypass database connection.

dbUtility.ts
import * as typeorm from 'typeorm';
import { AlreadyHasActiveConnectionError } from 'typeorm/error/AlreadyHasActiveConnectionError';
import { Driver } from 'typeorm/driver/Driver';
import { DriverFactory } from 'typeorm/driver/DriverFactory';
import { PostgresDriver } from 'typeorm/driver/postgres/PostgresDriver';
import { stub } from 'sinon';

// Load all of my entities from the index
import * as entities from './../../src/entity'

// I'm using postgres, so I just force the PG driver, but dont connect
class myFakeDBDriver extends PostgresDriver {
  async connect(): Promise<void> {}
}

// again, just overwriting the driver factory method to connect to fake PG
class myDriverFactory extends DriverFactory {
  create(connection: Connection): Driver {
    return new myFakeDBDriver(connection); 
  }
}

// Overwriting the connection driver with an accessor to ensure we're using the fake driver.
// Its readonly property value wouldn't allow me to reset this from a child constructor.
class Connection extends typeorm.Connection {
  _driver: typeorm.Driver;
  get driver(): typeorm.Driver {
    return this._driver;
  }
  set driver(options) {
    this._driver = new myDriverFactory().create(this);
  }
}

export class ConnectionManager extends typeorm.ConnectionManager {

  // Complete copy from ConnectionManager.connect, but now the Connection class uses the fake driver.
  create(options: typeorm.ConnectionOptions): Connection {
    // check if such connection is already registered
    const existConnection = this.connections.find(connection => connection.name === (options.name || "default"));
    if (existConnection) {
      // if connection is registered and its not closed then throw an error
      if (existConnection.isConnected)
        throw new AlreadyHasActiveConnectionError(options.name || "default");
      // if its registered but closed then simply remove it from the manager
      this.connections.splice(this.connections.indexOf(existConnection), 1);
    }
    // create a new connection
    const connection = new Connection(options);
    this.connections.push(connection);
    return connection;
  }
}

// Stubbing out the createConnetion method to ensure that we use our class overrides.
const createConnection = stub(typeorm, 'createConnection').callsFake(async function (optionsOrName:typeorm.ConnectionOptions) {
  const connectionName = typeof optionsOrName === "string" ? optionsOrName : "default";
  const options = optionsOrName instanceof Object ? optionsOrName : await typeorm.getConnectionOptions(connectionName);
  return new ConnectionManager().create(options).connect();
});

export default createConnection({
  name: 'test',
  type: 'postgres',
  entities: Object.values(entities)
} as typeorm.ConnectionOptions);

Test File

This file imports dbUtility and then sets a custom repository.
Missing from the example is my test to create an entity from the repository, which worked.

generic.spec.ts
import 'source-map-support/register';
import * as chai from 'chai';
import * as sinonChai from 'sinon-chai';
import { ItemRepository } from './../src/repository'
import StubConnection from './dbUtility';

chai.use(sinonChai);

const { expect } = chai;

describe('Connection', async () => {

  let connection, Item;

  before(async () => {
    connection = await StubConnection;
    Item = connection.getCustomRepository(ItemRepository);
  })

  it('...', async () => {
    // Do some stuff with Item
  })

})

@rightisleft
Copy link

So this thread dates back to 2017. Has there been any consensus from within the project on best practices? I see a lot of different partially viable approaches.

@pleerock
Copy link
Member

pleerock commented Feb 3, 2019

@rightisleft not yet, we still collecting feedback :) Think we can already find a good thoughts in there :)

@Namchee
Copy link

Namchee commented Jan 4, 2020

I know this is old, but for anyone looking for mongo memory server, check this out

@seandearnaley
Copy link

seandearnaley commented Jan 27, 2020

I ran into this issue, did some work on the subject and wrote an article about it with some solutions for testing a TypeScript+Apollo GraphQL+TypeORM server using Docker+GitHub Actions, may be helpful for some users here: published on gitconnected

@iamdevlinph
Copy link

After banging my head against this for a few days I ended up writing a small "utility" file that bypasses the DB connection via class overrides.

I haven't fully tested it but I've so far been able to create entities and run custom repository methods.

I plan on extending the utility to import raw JSON fixtures into memory.

I'm hoping that I dont run into too many issues with this, as it seems promising so far.
... Even better would be proper unit test utilities built into TypeORM!

Here are the files

dbUtility.ts

This file stubs out the createConnection method, and overrides core class methods to bypass database connection.

dbUtility.ts

import * as typeorm from 'typeorm';
import { AlreadyHasActiveConnectionError } from 'typeorm/error/AlreadyHasActiveConnectionError';
import { Driver } from 'typeorm/driver/Driver';
import { DriverFactory } from 'typeorm/driver/DriverFactory';
import { PostgresDriver } from 'typeorm/driver/postgres/PostgresDriver';
import { stub } from 'sinon';

// Load all of my entities from the index
import * as entities from './../../src/entity'

// I'm using postgres, so I just force the PG driver, but dont connect
class myFakeDBDriver extends PostgresDriver {
  async connect(): Promise<void> {}
}

// again, just overwriting the driver factory method to connect to fake PG
class myDriverFactory extends DriverFactory {
  create(connection: Connection): Driver {
    return new myFakeDBDriver(connection); 
  }
}

// Overwriting the connection driver with an accessor to ensure we're using the fake driver.
// Its readonly property value wouldn't allow me to reset this from a child constructor.
class Connection extends typeorm.Connection {
  _driver: typeorm.Driver;
  get driver(): typeorm.Driver {
    return this._driver;
  }
  set driver(options) {
    this._driver = new myDriverFactory().create(this);
  }
}

export class ConnectionManager extends typeorm.ConnectionManager {

  // Complete copy from ConnectionManager.connect, but now the Connection class uses the fake driver.
  create(options: typeorm.ConnectionOptions): Connection {
    // check if such connection is already registered
    const existConnection = this.connections.find(connection => connection.name === (options.name || "default"));
    if (existConnection) {
      // if connection is registered and its not closed then throw an error
      if (existConnection.isConnected)
        throw new AlreadyHasActiveConnectionError(options.name || "default");
      // if its registered but closed then simply remove it from the manager
      this.connections.splice(this.connections.indexOf(existConnection), 1);
    }
    // create a new connection
    const connection = new Connection(options);
    this.connections.push(connection);
    return connection;
  }
}

// Stubbing out the createConnetion method to ensure that we use our class overrides.
const createConnection = stub(typeorm, 'createConnection').callsFake(async function (optionsOrName:typeorm.ConnectionOptions) {
  const connectionName = typeof optionsOrName === "string" ? optionsOrName : "default";
  const options = optionsOrName instanceof Object ? optionsOrName : await typeorm.getConnectionOptions(connectionName);
  return new ConnectionManager().create(options).connect();
});

export default createConnection({
  name: 'test',
  type: 'postgres',
  entities: Object.values(entities)
} as typeorm.ConnectionOptions);

Test File

This file imports dbUtility and then sets a custom repository.
Missing from the example is my test to create an entity from the repository, which worked.

generic.spec.ts

import 'source-map-support/register';
import * as chai from 'chai';
import * as sinonChai from 'sinon-chai';
import { ItemRepository } from './../src/repository'
import StubConnection from './dbUtility';

chai.use(sinonChai);

const { expect } = chai;

describe('Connection', async () => {

  let connection, Item;

  before(async () => {
    connection = await StubConnection;
    Item = connection.getCustomRepository(ItemRepository);
  })

  it('...', async () => {
    // Do some stuff with Item
  })

})

This is the best lead I have but when I tried it I'm getting ConnectionIsNotSetError: Connection with postgres database is not established. Check connection configuration.

@nataneb32
Copy link

How make fast test?

I was think about it and the easy way is going to be a memory database, but they slow quite a bit the test suite. So you can separete all the interactions with the database in classes and use memory database to test them. When you test another thing that calls those classes, you should mock them. So almost all of your test suites are going to test your own code.

It should be cool to have something in typeorm, that make mocking it easier. Because there are multiples ways to do the samething and anyone in their right mind would try to mock all the ways.

@nahtnam
Copy link

nahtnam commented May 4, 2020

Sorry to bring up an old thread. One thing I do with knexjs is start a transaction globally before each test and then rollback at the end of the test. It looks something like this:

const { transaction, Model } = require('objection');

let knex;
let trx;

beforeEach(async () => {
  knex = Model.knex();
  trx = await transaction.start(knex);
  Model.knex(trx);
});

afterEach(async () => {
  await trx.rollback();
  Model.knex(knex);
});

afterAll(async () => {
  await knex.destroy();
});

Is it possible to do something like this with typeORM?

@arekbal
Copy link

arekbal commented Jun 9, 2020

Reverting transaction strategy makes it integration test, not a unit/isolated test. They have their place, but these are expensive in terms of resources. It leads to smaller suites, tests not being run by developers and long deploys.
True custom repository(not CustomRepository) wrappers, seem to be the strongest approach to mocks. Ignore foreign key constraints and some of other problems related to doing Map<> based repo mocks.

@HunderlineK
Copy link

HunderlineK commented Jun 17, 2020

The recommended approach in Java land is using actual databases:
https://www.testcontainers.org/

I'd personally never rely on unit tests on an ORM, even if it was possible; what if there's a bug in the ORM that only surfaces in your specific use cases? You can't catch those by mocking your database calls. Dry running is also scary and little different from using an actual database excluding the startup time (which should diminish as a factor as your tests grow anyway)

@arekbal
Copy link

arekbal commented Jun 19, 2020

@HunderlineK That's what I suggest to do and we do those for some extra checks. But integration tests are heavy in many dimensions.

I'd personally never rely on unit tests on an ORM, even if it was possible; what if there's a bug in the ORM that only surfaces in your specific use cases?

You can apply this statement to any library you use, would that be Axiom, or int8l lib... whatever. There will always be a mixture of strictly "your" code, std-lib code and some of external libs code inside your unit tests.

You want to have these cases covered as much as it makes sense. Not too grained, run fast so developers can run them with confidence, and not expensive to maintain.

@mffap
Copy link

mffap commented Jun 29, 2020

I was also stuck getting unit tests with mocked typeorm and Jest to work. My fallback option was to use a in-memory sqlite db, but as mentioned here, this is not an option for everyone.

In my case I want to test a Service that serves financial accounts stored in a database. The Service creates an account-repository with typeorm getRepository() function. I then use .find(), .save() etc. to interact with the repository and finally return the account (or undefined).

It might not be the best solution, but afer reading through a lot of comments and posts, this might help someone, who is new to typeorm like me, to get started. Also checkout the comments here from @lytc and @brdk .

//<app>/src/services/AccountService.ts

import { Account } from "./Account";
import { getRepository } from "typeorm";

export class AccountService {

  private accountRepo = getRepository(Account);

  public async getById(id : number): Promise<Account | undefined> {
    const account = await this.accountRepo.findOne(id)
    return account
  }

}

I have created a manual mock for typeorm, as follows. There might be a prettier way, but it works for me at the moment.

// <app>/__mocks__/typeorm.ts

export const PrimaryGeneratedColumn = jest.fn()
export const Column = jest.fn()
export const Entity = jest.fn()

export const getRepository = jest.fn().mockReturnValue({
    findOne: jest.fn() // return value will be set in the test
  })

My test are then defined like this:

//<app>/src/services/AccountService.test.ts
import { AccountService } from "./AccountService";
import { getRepository } from 'typeorm'
import {mocked} from 'ts-jest/utils'
import faker from 'faker'

jest.mock('typeorm')

describe('AccountService', () => {
  //https://github.com/kulshekhar/ts-jest/blob/master/docs/user/test-helpers.md
  const mockedGetRepo = mocked(getRepository(<jest.Mock>{}))

  beforeEach(() => {
    mockedGetRepo.findOne.mockClear()
  })

  test('getById', async () => {
    //Arrange
    const fakeAccount = {
      id: 123,
      iban: faker.finance.iban(),
      name: faker.finance.accountName()
    }
    mockedGetRepo.findOne.mockResolvedValue(fakeAccount) // return value
    //Act
    const accountService = new AccountService()
    const account = await accountService.getById(fakeAccount.id)
    //Assess
    expect(account).toEqual(fakeAccount)
  })

@adabalasuresh05
Copy link

Hi @YegorZaremba ,
https://github.com/YegorZaremba/typeorm-mock-unit-testing-example/tree/master/test/unit/mocha
I followed the your sample code from the above link but i got the default connection error,can you please help me

@AnubhavUjjawal
Copy link

AnubhavUjjawal commented Mar 23, 2021

I already have mock responses for repository methods which I am gonna use and I don't want Typeorm to connect to DB when running unit tests.

This solved the issue:

jest.spyOn(Connection.prototype, "connect").mockReturnThis()

@vitordelfino
Copy link

I'am using Jest manual mock, to "inject" a fake repositery in my class Service

I wrote an article in brazilian portuguese: https://dev.to/vitordelfino/escrevendo-testes-com-jest-supertest-1ed

Repository: https://github.com/vitordelfino/tutorial-node

Now, I'am studying type + typeorm + routing controller and trying to find a way to mock my services

Sorry for my english

@mikhail-angelov
Copy link

I faced with the same problem once I worked on NestJS project
so, I have to make small lib https://github.com/mikhail-angelov/nestjs-db-unit which helps me replace postgres with in memory sqlite DB to make integration tests

@kelvis-jobsity
Copy link

I'm using mocha, chai, and moq.js and it works like a charm for me:

My service:

import { Service } from "typedi";
import { Repository } from "typeorm";
import { InjectRepository } from "typeorm-typedi-extensions";
import { User} from "../shared/entities/user/entity/user.entity";
import { PaginatedDataExtractor } from "../shared/etl-process/data-extractor";
import { PaginatedData } from "../shared/etl-process/paginated-data";

@Service()
export class Extractor implements PaginatedDataExtractor<User[]> {

    constructor(@InjectRepository(User) private repository: Repository<User>) { }

    async extractPage(ids: number[], page = 1, pageSize = 1000): Promise<PaginatedData<User[]>> {

        try {
            console.log(`Extracting page ${page} with max size of ${pageSize}`);

            let query = this.repository
                .createQueryBuilder('users')
                .leftJoinAndSelect('users.status', 'status')
                .leftJoinAndSelect('users.type', 'type')
                .leftJoinAndSelect('users.photos', 'photo');

            if (ids.length > 0)
                query = query.whereInIds(ids)

            const [list, count] = await query
                .orderBy('users.id', 'ASC')
                .skip((page - 1) * pageSize)
                .take(pageSize)
                .getManyAndCount();

            const numPages = Math.ceil(count / pageSize)

            console.log(`Extracted page ${page}/${numPages} with ${list.length} itens | Total: ${count}`);

            const hasNext = page < numPages;
            const nextPage = hasNext ? page + 1 : page;

            console.log(`${hasNext ? `Next page is ${nextPage}` : 'No next page'}`);
            return {
                page: list,
                hasNext: hasNext,
                nextPage: nextPage
            }
        } catch (error) {
            console.error(error);
            throw error;
        }


    }
}

My service unit tests:

import 'mocha';
import { should } from 'chai';
import { Mock, It, Times } from 'moq.ts';
import { Repository, SelectQueryBuilder } from 'typeorm';
import { Extractor } from '../../../src/apps/etl/extractor';
import { User } from '../../../src/apps/shared/entities/user/entity/user.entity';
import { PaginatedData } from '../../../src/apps/shared/etl-process/paginated-data';
should();

describe('## Extractor ##', () => {

    const fakeDataArr = Array.from(Array(20), (x, index) => ({ id: index + 1 } as User));

    const queryBuilderMock = new Mock<SelectQueryBuilder<User>>();

    queryBuilderMock.setup(i => i.leftJoinAndSelect(It.IsAny<string>(), It.IsAny<string>()))
        .returns(queryBuilderMock.object());

    queryBuilderMock.setup(i => i.whereInIds(It.IsAny<number[]>()))
        .returns(queryBuilderMock.object());

    queryBuilderMock.setup(i => i.orderBy(It.IsAny<string>(), It.IsAny<string>()))
        .returns(queryBuilderMock.object());

    queryBuilderMock.setup(i => i.skip(It.IsAny<number>()))
        .returns(queryBuilderMock.object());

    queryBuilderMock.setup(i => i.take(It.IsAny<number>()))
        .returns(queryBuilderMock.object());

    queryBuilderMock.setup(i => i.getManyAndCount())
        .returns(Promise.resolve([fakeDataArr.slice(0, 10), fakeDataArr.length]));

    const repositoryMock = new Mock<Repository<User>>()
        .setup(i => i.createQueryBuilder(It.IsAny<string>()))
        .returns(queryBuilderMock.object());

    describe('When extractPage is called', () => {

        const extractor = new Extractor(repositoryMock.object());
        const ids = [1, 2, 3];
        const page = 1;
        const pageSize = 10;        
        let result: PaginatedData<User[]>;
        
        before(async () => {
            result = await extractor.extractPage(ids, page, pageSize);
        });

        it('Should create query builder with users alias', () => {
            repositoryMock.verify(x => x.createQueryBuilder(It.Is<string>(v => v === 'users')), Times.Once());
        });

        it('Should left join with status', () => {
            queryBuilderMock.verify(x => x.leftJoinAndSelect(
                It.Is<string>(v => v === 'users.status'),
                It.Is<string>(v => v === 'status')),
                Times.Once()
            );
        });

        it('Should left join with type', () => {
            queryBuilderMock.verify(x => x.leftJoinAndSelect(
                It.Is<string>(v => v === 'users.type'),
                It.Is<string>(v => v === 'type')),
                Times.Once()
            );
        });

        it('Should left join with photos', () => {
            queryBuilderMock.verify(x => x.leftJoinAndSelect(
                It.Is<string>(v => v === 'users.photos'),
                It.Is<string>(v => v === 'photo')),
                Times.Once()
            );
        });        

        it('Should order by id ASC', () => {
            queryBuilderMock.verify(x => x.orderBy(
                It.Is<string>(v => v === 'users.id'),
                It.Is<string>(v => v === 'ASC')),
                Times.Once()
            );
        });

        it('Should skip the right number of itens', () => {
            queryBuilderMock.verify(x => x.skip(
                It.Is<number>(v => v === (page - 1) * pageSize)),
                Times.Once()
            );
        });

        it('Should take the right number of itens', () => {
            queryBuilderMock.verify(x => x.take(
                It.Is<number>(v => v === pageSize)),
                Times.Once()
            );
        });

        it('Should call getManyAndCount once', () => {
            queryBuilderMock.verify(x => x.getManyAndCount(), Times.Once());
        });

        it('Should return only the page size', () => {
            result.page.length.should.be.equal(pageSize);
        });

        it('Should hasNext be true', () => {
            result.hasNext.should.be.equals(true);
        });

        it('Should nextPage be 2', () => {
            result.nextPage.should.be.equals(2);
        });
    });
});

@llaenowyd
Copy link

I have a lot of mocked out unit tests without much trouble, but found my way to this thread looking to create an integration test that works without a database - in this case we're using postgres and so my thought was it would make a nice test setup to spy on the SQL statements produced by the pg driver without connecting. One of my teammates is more sceptical about using an ORM, and he's knowledgeable about postgres SQL, so having a lightweight in-memory test from fake HTTP request all the way to verifying SQL statements sounds perfect, and, seems like it ought to be feasible. Perhaps not as easy as I'd hoped.

@mapennig
Copy link

mapennig commented Oct 5, 2021

That should to the trick!

import { Connection } from "typeorm";

const createConnectionStub = sinon.stub(Connection.prototype, "connect");

@pleerock pleerock removed this from the 1.0.0 milestone Nov 10, 2021
@NikolaLukic1
Copy link

Did anyone encounter this issue using DataSource implementation?

@Thore1954
Copy link

I agree that using in memory sqlite/sql.js is too much overhead for unit testing but I don't think mocking typeorm is a good idea either since as a general rule you shouldn't mock the types you don't own.

I really like how you intercept http requests and return ad-hoc results using nock especially how you match the requests:

nock('http://example.com')
  .get('/users')
  .query({ name: 'pedro', surname: 'teixeira' })
  .reply(200, { results: [{ id: 'pgte' }] })

I'd love typeorm to provide a way to intercept queries without a database. There's already a strong foundation called QueryBuilder which can be reused to match queries:

const scope = await AppDataSource.getRepository(Photo)
    .interceptQueries("photo")
    .where("photo.isPublished = true")
    .andWhere("(photo.name = :photoName OR photo.name = :bearName)")
    .setParameters({ photoName: "My", bearName: "Mishka" })
    .returnMany(photos)

@grantvanhorn
Copy link

For others who are looking for in memory solutions but don't want to deal with type mismatching between SQLite and Postgres, I am using this in memory Postgres package: https://github.com/oguimbal/pg-mem

Here is an example of how to set pg-mem up: https://gist.github.com/navjotahuja92/f51601b17fb248cf4727b5650d945607

@clintonb
Copy link
Contributor

clintonb commented Jun 9, 2023

I'm in a different situation than the majority of you. I want to create instances of entities via dataSource.getRepository(MyModel).create() without connecting to the DB. My tests are setup to operate on in-memory instances since they are unit tests; but, I don't want to connect to the database when I don't intend to actually query the database.

I have discovered that calling DataSource.buildMetadatas() works after instantiating a DataSource. The caveat is I must sub-class DataSource to expose this method.

@hinogi
Copy link

hinogi commented Jun 30, 2023

Wouldn't it be better to just mock all of TypeORM away? You are not testing TypeORM, you just want a valid or invalid database result given back to the function call, I guess.
Given the example:

jest.mock("typeorm", () => ({
    __esModule: true,
    getRepository: jest.fn().mockReturnThis(),
    interceptQueries: jest.fn().mockReturnThis(),
    where: jest.fn().mockReturnThis(),
    andWhere: jest.fn().mockReturnThis(),
    setParameters: jest.fn().mockReturnThis(),
    returnMany: jest.fn().mockReturnValue(() => []),
}));

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests