Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

File: TAR archive reader #2268

Merged
merged 7 commits into from
Apr 23, 2020
Merged

File: TAR archive reader #2268

merged 7 commits into from
Apr 23, 2020

Conversation

ennru
Copy link
Member

@ennru ennru commented Apr 14, 2020

Add reading of tar-formatted streams of ByteStrings.

Inspired by the recent addition of writing tar in #2241 by @choffmeister.

  • Add TarReaderStage
  • Tar header parsing in TarArchiveEntry
  • Improve ByteString handling in TarArchiveEntry header creation
  • Move tar tests into own spec

TODO

  • verify use of SubSourceOutlet
  • documentation

@ennru ennru marked this pull request as ready for review April 19, 2020 16:30
Copy link
Member

@raboof raboof left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Always pretty hard to see if all the pulling/pushing aligns neatly, but looks good AFAICS. The low-level tar logic remained pretty neat.

file/src/test/scala/docs/scaladsl/TarArchiveSpec.scala Outdated Show resolved Hide resolved
@@ -27,4 +27,10 @@ object Archive {
def tar(): Flow[(TarArchiveMetadata, Source[ByteString, _]), ByteString, NotUsed] =
TarArchiveManager.tarFlow()

/**
* Parse incoming `ByteString`s into tar file entries and sources for the file contents.
* The file contents sources MUST be consumed to progress reading the file.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, this is kinda dangerous - uses like bytes.via(tarReader).take(100).foreach(_.runWith(Sink.ignore)) could stall the stream.

I seem to remember we have similar problems in processing multipart file uploads in Akka HTTP, but can't find a reference right now.

I don't see how we could make an API that avoids this, though, so perhaps this comment is sufficient.

@ennru ennru requested a review from raboof April 23, 2020 07:22
@ennru ennru added this to the 2.0.0 milestone Apr 23, 2020
@ennru ennru merged commit 62918fd into akka:master Apr 23, 2020
@ennru ennru deleted the tar-reader branch April 23, 2020 08:24
@mkurz
Copy link
Contributor

mkurz commented May 1, 2020

Hey @ennru, given your work on this pull request here, any chance you also add a ZIP archive reader?

@ennru
Copy link
Member Author

ennru commented May 4, 2020

I have no plans for that at the moment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants