read huge data

[fabiooshiro]

When I tried to read the 1,28 GB file from
http://svs.aids.gov.br/dantps/centrais-de-conteudos/dados-abertos/sim/
http://svs.aids.gov.br/dantps/centrais-de-conteudos/dados-abertos/sim/DO21OPEN.zip

my computer cant open that file, so I did a change to allow an optional handler to read the data without keep all records in memory.

Example:
https://github.com/fabiooshiro/covid

[yortus]

Hi @fabiooshiro, thanks for the PR! The current API is already designed to read very large files by reading records in batches. I've outlined an example below that should hopefully work fine for your case. As such I don't think the changes in this PR are necessary, so I'll be closing this. If you have any problems please feel free to post back here or create a new issue.

const BATCH_SIZE = 1000;
(async () => {
    const dbf = await DBFFile.open('./DO21OPEN.dbf');
    while (true) {
        // read the next batch of records
        const records = await dbf.readRecords(BATCH_SIZE);

        // exit the while loop when no more records are read
        if (records.length === 0) break; // done

        // process each record in this batch
        for (const record of records) {
            // process each record here...
        }
    }
})();

[fabiooshiro]

Now I understand.
Thank you!