-
Notifications
You must be signed in to change notification settings - Fork 16
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unable to decompress large file #103
Comments
import path from "path";
import fs from "fs-extra";
import unrar from "node-unrar-js";
export default class RAR {
filePath: string;
constructor(_filePath: string) {
this.filePath = _filePath;
}
readStreamFile = (_filePath: string): Promise<Buffer> =>
new Promise((resolve, reject) => {
let buff = Buffer.from([]);
const readStream = fs.createReadStream(_filePath, {
highWaterMark: 128 * 1024 * 1024,
});
readStream.on("data", (nextBuff) => {
buff = Buffer.concat(
[buff, nextBuff as Buffer],
buff.length + nextBuff.length
);
});
readStream.on("end", () => resolve(buff));
readStream.on("error", (error) => reject(error));
});
extractAllTo = async (targetPath: string, options?: IOptions) => {
try {
const buff = await this.readStreamFile(this.filePath);
const extractor = await unrar.createExtractorFromData({
data: Uint8Array.from(buff).buffer,
password: options?.password,
});
const { files } = extractor.extract({
files: (fileHeader) => {
if (!fileHeader.flags.directory) {
if (/\.txt$/.test(fileHeader.name)) {
return true;
} else {
return false;
}
}
return false;
},
});
try {
for (const { fileHeader, extraction } of files) {
try {
const isDirectory = fileHeader.flags.directory;
if (isDirectory) {
fs.ensureDirSync(path.join(targetPath, fileHeader.name));
} else {
fs.outputFileSync(
path.join(targetPath, fileHeader.name),
extraction
);
}
} catch {}
}
} catch {}
return true;
} catch {
return false;
}
};
}
export interface IOptions {
password?: string;
} |
Quoting doc
I guess it's because the file is too big to fit in the memory, you can try to download the file to the file system, then use |
Your ideal worked with me, Thanks for that. |
Why the error in the call stack is thrown from |
It's still wrapped by a third-part dependency node_modules/request, can you run it without pm2 to see how the error looks like? |
I'll run it again in developer mode, I'll be right back, don't go anywhere bro, I need it urgently. |
Why wrapped by Vite this time? can you just run a small example by Like
and use node to run it directly |
I tried what you said, and it works, I think it's a problem with my code |
I am extracting my data file about 5gb, I tried to filter out the necessary files, but after a while of unzipping, I get the following message.
The text was updated successfully, but these errors were encountered: