listObjects lack retry mechanism

[clojurians-org]

i fetch huge file size from server, it will contains 3 * 10^5 file number.

when i run listObjects, it report Request Timeout.

[harshavardhana]

Please provide more detailed reproducible steps, along with sample code.

[clojurians-org]

i pull a lot of file from the server for backup operation(about 300K).
the server become unstable status and response with http timeout.

i fix the issue by add simple retry operation in list conduit, and add http managerSetting timeout duration.
i just hardcode for default retry policy from hackage retry library
the getObject retry can be process in my call executable, so it's not a problem.

Network/Minio/ListOps.hs

...
-- | List objects in a bucket matching the given prefix. If recurse is
-- set to True objects matching prefix are recursively listed.
listObjectsV1 :: Bucket -> Maybe Text -> Bool
              -> C.ConduitM () ObjectInfo Minio ()
listObjectsV1 bucket prefix recurse = loop Nothing
  where
    loop :: Maybe Text -> C.ConduitM () ObjectInfo Minio ()
    loop nextMarker = do
      let delimiter = bool (Just "/") Nothing recurse
      let listObjectsMaybe = fmap Just (listObjectsV1' bucket prefix nextMarker delimiter Nothing)
                                `catch` \(e :: SomeException) -> liftIO (putStrLn (show e)) >> return Nothing

      res <- lift $ fmap fromJust $ retrying def  (const $ return . isNothing) (const listObjectsMaybe)
      liftIO $ putStrLn $ "[debug] fetching dir ..."
      CL.sourceList $ lorObjects' res
      when (lorHasMore' res) $
        loop (lorNextMarker res)

[clojurians-org]

i upload the related code to github:
but i don't have tons of data for reproducible, as i just write simple script to use.
https://github.com/clojurians-org/haskell-example/blob/master/minio-migration/src/Main.hs

[donatello]

Thanks for the report, we will take a look.

[donatello]

@clojurians-org We understand the problem - the library does not implement any retry functionality for request timeouts. We will add this feature. Thanks again for the report!

[donatello]

@clojurians-org Could you try out version 1.3.0? It contains a retry mechanism.

[clojurians-org]

yes, it's work now.
BTW, i use the folloing code to build MinioConn for speicifying timeout.

the ConnectInfo(..) data constructor didn't export, the connectIsSecure can't be used.

mkMC :: ConnectInfo -> Text -> Text -> Int -> IO MinioConn
mkMC ci accessKey secretKey timeout = do
  let ci' =  ci & setCreds (Credentials accessKey secretKey) & setRegion "us-east-1"
  let timeoutSeconds = 1000 * 1000 * timeout
  let settings | connectIsSecure ci' = tlsManagerSettings
               | otherwise = defaultManagerSettings
  mgr <- newManager settings
           { managerResponseTimeout = responseTimeoutMicro timeoutSeconds }
  mkMinioConn  ci' mgr

[donatello]

the ConnectInfo(..) data constructor didn't export, the connectIsSecure can't be used.

@clojurians-org We have a fix here #121 - we will make a minor release after it is merged.

[donatello]

@clojurians-org Version 1.3.1 has change #121 and should solve your problem.