Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

added: fileSourceByteString #1503

Merged
merged 10 commits into from May 4, 2018

Conversation

Projects
None yet
2 participants
@ncaq
Copy link
Contributor

ncaq commented Apr 17, 2018

This function is to get FileInfo raw body.

Conduit type is hard for begginer user.

Actually I was not sure when I was a Haskell beginner.

ByteString raw body of FileInfo is need to upload file to S3.
This function running our product.
I send it to upstream.

Before submitting your PR, check that you've:

After submitting your PR:

  • Update the Changelog.md file with a link to your PR
  • Check that CI passes (or if it fails, for reasons unrelated to your change, like CI timeouts)

ncaq added some commits Apr 17, 2018

added: fileSourceByteString
This function is to get `FileInfo` raw body.
--
-- @since 1.6.4
fileSourceByteString :: MonadResource m => FileInfo -> m S.ByteString
fileSourceByteString fileInfo = fileSource fileInfo `connect` CL.foldMap id

This comment has been minimized.

Copy link
@snoyberg

snoyberg Apr 17, 2018

Member

There are a few problems with this implementation:

  1. CL.foldMap id has quadratic complexity. It would be better to use something like sinkLazy
  2. It's more common to use runConduit $ src .| sink

How about:

runConduit $ fileSource fileInfo .| Conduit.

This comment has been minimized.

Copy link
@ncaq

ncaq Apr 17, 2018

Author Contributor

I use runConduit.
But fileSource function result strict ByteString.
So I think can't use sinkLazy.
Do you think?

This comment has been minimized.

Copy link
@snoyberg

snoyberg Apr 17, 2018

Member

You'd have to use Data.ByteString.toStrict, but that's more efficient than using foldMap.

@@ -1360,6 +1361,16 @@ rawRequestBody = do
fileSource :: MonadResource m => FileInfo -> ConduitT () S.ByteString m ()
fileSource = transPipe liftResourceT . fileSourceRaw

-- | Strict `ByteString` body from `FileInfo`.
-- This function blocking while read file.

This comment has been minimized.

Copy link
@snoyberg

snoyberg Apr 17, 2018

Member

Can you reword this comment to something like:

Extract a strict `ByteString` body from a `FileInfo`.

This function will block while reading the file.
@ncaq

This comment has been minimized.

Copy link
Contributor Author

ncaq commented Apr 17, 2018

@snoyberg I think that sinkLazy can't use this case.
Do you think?

@ncaq

This comment has been minimized.

Copy link
Contributor Author

ncaq commented Apr 17, 2018

@snoyberg
I use sinkLazy.
However, Maybe fileSourceByteString return Lazy ByteString is great than Strict?

@snoyberg

This comment has been minimized.

Copy link
Member

snoyberg commented Apr 18, 2018

Returning a lazy ByteString instead is a good idea, though then it should be documented to clarify that the value will still be strictly read (as opposed to lazy I/O).

@ncaq

This comment has been minimized.

Copy link
Contributor Author

ncaq commented Apr 24, 2018

@snoyberg
I think again.
fileSource is strict, so fileSourceByteString should be strict.

So, I use foldC. Do you think about?

@snoyberg

This comment has been minimized.

Copy link
Member

snoyberg commented Apr 24, 2018

foldC has the same quadratic behavior issues as foldMapC.

@ncaq

This comment has been minimized.

Copy link
Contributor Author

ncaq commented Apr 24, 2018

@snoyberg
Sorry, What are "complexity" mean code clearing or performance?
I understood to code clearing.
So, I try to write clear code.
But, Are you speak performance?

@snoyberg

This comment has been minimized.

Copy link
Member

snoyberg commented Apr 24, 2018

modified: use sinkLazy and toStrict
Because performance problem.
@ncaq

This comment has been minimized.

Copy link
Contributor Author

ncaq commented May 1, 2018

@snoyberg I rewrite.

@snoyberg
Copy link
Member

snoyberg left a comment

Thanks!

@snoyberg snoyberg merged commit 5861357 into yesodweb:master May 4, 2018

0 of 2 checks passed

continuous-integration/appveyor/pr AppVeyor build failed
Details
continuous-integration/travis-ci/pr The Travis CI build failed
Details

@ncaq ncaq deleted the ncaq:add-file-source-bytes branch May 4, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.