Unable to modify files during packing due to "size mismatch"

[abresas]

The readme file mentions using mapStream during pack.entry, but any transformation on the stream that alters the size of the contents results in "size mismatch" error, thrown from tar-stream here https://github.com/mafintosh/tar-stream/blob/bbb3e91a44fde7bdb3d179e27afdaa4b08fa74ac/pack.js#L174.

The error is silent. Unless debugging logs are added to onnextentry function of tar-fs, the error manifests as the stream simply freezing.

Is there a way to get around this?

[benfoxall]

This tripped me up for a while today. @abresas did you find a work around?

[mafintosh]

we should make it emit an error in this case for sure. if you actually wanna change the size you could update the header as well

[benfoxall]

Thanks - that lets me get around it with:

  mapStream: (stream, header) => {
    var size = 0
    return stream
      .pipe(myTransform)
        .on('data', chunk => size += chunk.length)
        .on('end', _ => header.size = size)
  }

Should this be handled by tar-fs? I imagine that most of the time people are providing mapStream, they'll be changing the size of the file. Would you want a PR raised?

[mafintosh]

@benfoxall unfortunately you need to update header.size before mapping the stream because of a tar limitation so we can't really make the module handle it for you

[crysislinux]

@benfoxall the workaround don't work in fact. the error is gone. but the size is still not changed. the transformed file will have some strange chars.

[dodtsair]

Ran into this today. Was using a regular expression to find a semver that was static. In this case 0.0.0 and replace it with a dynamic version in this case 1.0.31. The single character change took me an hour to identify.

The tar limitation, the file header and size is written first, then the file data. In this case the map function would run and set the file header information, then the mapStream function would run transforming the bytes as they are laid down in the tarball.

Would it be possible to watch the bytes written and seek back to the file header and fix up the length?

[payitforwardnow]

I appreciate your time and the time you spent on this project; however, it's been over two years since this issue was first reported and while I understand you can't change the tar format, you can help prevent others from wasting hours and hours of their valuable time if you would just take 5 minutes of your valuable time to put something in the documentation.

Can someone PLEASE update the README.md where you describe the mapStream function? Where you have

incase you wanna modify the input/output file streams

How about changing it to

incase you wanna modify the input/output file streams (NOTE: the size of the data must not change so compression/decompression transforms won't work, for example)

Is that an unreasonable request? 5 minutes vs. hours and hours and hours of wasted time people will never get back!! Sorry rant over....

Ok so mapStream allows you to transform the data; however, as I understand it, there is a limitation (a big one IMHO) that the size of the data can't change due to the limitations on the tar format. Curious what transforms are useful other than the obvious encryption transforms. I guess if I want to continue to use tar-fs I will have no choice but to compress all the files first then add them to the tar one at a time. Given the shear amount of time I've spent today trying to resolve this issue I've just made up my mind to create a new intelligent bundle format that allows true on-the-fly transforms allowing each individual file to be compressed using the absolute best compression scheme (or none) on a per file basis as well as having the ability to use unique encryption keys on a per file basis. I can't tell you how disappointed I am in whats NOT available. Just when I thought tar-fs was going to work for me. Tons of tar stuff, zip stuff and crypto stuff out there but nothing that really pulls all of it together in way that is flexible, efficient and EASY TO USE!!! OK on the positive side I still have some hair left :)

tar format limitations aside, can someone PLEASE put something in the README.md to let people know (or remind them in case they are tar experts) that the transforms only work if the data size doesn't change? At the very least provide an error that can be seen and understood. In the meantime if anyone is interested in a flexible unique bundling format please let me know. I have the design and just need some feedback.

Thank you in advance for your time and consideration.

[payitforwardnow]

FYI - I found a solution to this using a different archive format. Not the developers fault. Unfortunately, the "tar" format has a lot of limitations (for my needs) so I decided to find/create a better solution. This project will will for a majority of people who need to create a "tar"; however, if "tar" is not necessary there are other alternatives. There is a way you can create an archive of an entire directory and filter, compress and/or encrypt "on-the-fly". One of the great things about this new solution is that the archive is also random access so you can retrieve individual files without having to read in the entire archive. Each file can be compressed based on file type (or not) and can be encrypted using any encryption scheme you want with its own unique encryption key (if you prefer). Anyway if anyone is interested please let me know. Thank you again for your time and consideration.

[mafintosh]

Will merge a PR that makes the readme better for this

[payitforwardnow]

Thank you very much!!! I'm sure it will save people a lot of time.

Your time and consideration is appreciated.

[knoxcard]

Close ticket?

[revanth0212]

This can also happen if you are writing into the same folder that is being tared. I changed my tar path and it worked just fine.