"ipfs.get(cid, { archive: false })" ignores the archive flag

[jurelik]

• Version:
  go-ipfs: 0.9.1
  js-ipfs: 0.11.0
  ipfs-http-client: 53.0.0

• Platform:
  Darwin jure.local 18.7.0 Darwin Kernel Version 18.7.0: Tue Jun 22 19:37:08 PDT 2021; root:xnu-4903.278.70~1/RELEASE_X86_64 x86_64

• Subsystem:
  Get

Severity:

Medium

Description:

I would like to get a folder without it being put into a tarball. This works in the go-ipfs CLI by not applying the -a flag, but going through either ipfs-core or ipfs-http-client+go-ipfs, the archive:false flag doesn't work. Instead, I always get a tar version of the folder.

Steps to reproduce the error:

const IPFS = require('ipfs-core');
const fsp = require('fs').promises;
const fs = require('fs');
const path = require('path');

const func = async () => {
  const ipfs = await IPFS.create()

  await fsp.mkdir(folderPath, { recursive: true });

  const stream = fs.createWriteStream(path.join(folderPath, folderName));
  for await (const file of ipfs.get(folderCID, { archive: false })) stream.write(file);
  stream.end();
}

func();

[jurelik]

I manged to solve the issue so I'm documenting it here in case anyone stumbles across a similar problem. This thread was a lot of help. As far as I understand, api/v0/get will always output a tar file, unlike the CLI command. So the only way to get around that is to use api/v0/cat instead and stream the output of that into a writeStream. The code above now looks like:

const fsp = require('fs').promises;
const fs = require('fs');
const path = require('path');
const app = {
  ipfs: null
}

const func = async () => {
  const ipfs = await IPFS.create()
  app.ipfs = ipfs;

  await fsp.mkdir(folderPath, { recursive: true });
  await iterateDirectory(folderPath, cid);

  console.log('success')
}

const iterateDirectory = async (downloadPath, cid) => {
  try {
    for await (const file of app.ipfs.ls(cid)) {
      if (file.type === 'file') {
        const stream = fs.createWriteStream(path.join(downloadPath, file.name));
        for await (const chunk of app.ipfs.cat(file.path)) stream.write(chunk);
        stream.end();
      }
      else if (file.type === 'dir') {
        const folderPath = path.join(downloadPath, file.name);
        await fsp.mkdir(folderPath, { recursive: true });
        await iterateDirectory(folderPath, file.path);
      }
    }
  }
  catch (err) {
    console.error(err)
  }
}

func();

What I'm doing is using the api/v0/ls to look at the contents of the "top level" folder. If file.type === 'file' I use api/v0/cat to stream to disk and if file.type === 'dir' I trigger the iterateDirectory function again with a new downloadPath & new "top level" folder cid.

Hope this helps but also please let me know if you know a better way to approach this!

[achingbrain]

Looks like you figured it out. Making repeated calls to ipfs.ls and ipfs.cat is one way to do it, though it involves multiple HTTP requests which may be inefficient if the directory structure is particularly large.

Another way to do it would be to unpack the tar stream and process the contents - this is how the ipfs get CLI command is implemented - see these lines in packages/ipfs-cli/src/get.js.