Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Ready] Fix _ObjectReader.seek(0, os.SEEK_END) case #242

Merged
merged 4 commits into from
Dec 23, 2021

Conversation

belltailjp
Copy link
Member

@belltailjp belltailjp commented Dec 22, 2021

It addresses #239

This PR fixes the inconsistent behavior of seek when opening a file on S3.

with pfio.v2.open_url('s3://<bucket>/xxxx', 'rb') as f:
    f.seek(0, os.SEEK_END)    # <------ It should move the cursor to the last byte of the stream, but it didn't

Fixing only the above case is enough to fix the issue reported in #239.
But for better transparency of pfio API with other types of filesystems, I updated the whole seek behavior to work identically to normal file-like objects returned by open (binary mode).

@belltailjp
Copy link
Member Author

/test

@pfn-ci-bot
Copy link

Successfully created a job for commit 27aa610:

1 similar comment
@pfn-ci-bot
Copy link

Successfully created a job for commit 27aa610:

@kuenishi kuenishi added the cat:bug Bug report or fix. label Dec 22, 2021
@kuenishi
Copy link
Member

/test Thanks for contribution! LGTM

@pfn-ci-bot
Copy link

Successfully created a job for commit 14d1dba:

1 similar comment
@pfn-ci-bot
Copy link

Successfully created a job for commit 14d1dba:

@kuenishi kuenishi added this to the 2.0.1 milestone Dec 22, 2021
@belltailjp belltailjp changed the title Fix _ObjectReader.seek(0, os.SEEK_END) case [WIP] Fix _ObjectReader.seek(0, os.SEEK_END) case Dec 22, 2021
@belltailjp
Copy link
Member Author

/test

@pfn-ci-bot
Copy link

Successfully created a job for commit 42b9d20:

@pfn-ci-bot
Copy link

  [UNAVAILABLE] failed to generate serial numbers
  2021-12-22 23:43:19.234414 database.go:348] transaction failed: API error 5 (datastore_v3: TIMEOUT): The datastore operation timed out, or the data was temporarily unavailable. entity groups: [(app=b~pfn-public-ci, kv, "9d344b6a8b151dacbf2144a2d66da50f4b9798ea4af0a044d8a1b5860d4d5ce2"),(app=b~pfn-public-ci, kv, "7137437ea36da4f5eece25816cb5bb03831ecd5ae3a077777df8ec1f368d95e6"),(app=b~pfn-public-ci, kv, "98c3e26365ea16d561df6af9199372b7e72b823c2ab9b9bb8d37326de9062c92"),(app=b~pfn-public-ci, kv, "ccc3a06c2ee72de1a6393b56961a2f898723b93d914b4363b7b6442f65a53dd6"),(app=b~pfn-public-ci, kv, "0e58d640d95b894bf6b6a01104f2049f818affd335ba0c18d422093c6dbdb2fa"),(app=b~pfn-public-ci, kv, "166e13729a2131747e625476b73d28deac37167f5a309cdd91ca108e1e4f3295"),(app=b~pfn-public-ci, kv, "bfdba543a4ae79ea832decce9af412ec5d366d342789ff9d3b8954d0426df740")]
  2021-12-22 23:43:19.234483 generator.go:156] failed to generate serial numbers
  
  Stack trace:
    github.com/pfnet/flexci/internal/frontend/database.transactionOnce (database.go:348)
    github.com/pfnet/flexci/internal/frontend/database.Transaction (database.go:300)
    github.com/pfnet/flexci/internal/frontend/util/idutil.GenerateIDs (generator.go:93)
    github.com/pfnet/flexci/internal/frontend/util/idutil.GenerateID (generator.go:164)
    github.com/pfnet/flexci/internal/frontend/util/idutil.LazyGenerateID.func1 (generator.go:190)
    github.com/pfnet/flexci/internal/frontend/util/idutil.(*LazyIDGenerator).Do.func1 (generator.go:180)
    sync.(*Once).Do (once.go:44)
    github.com/pfnet/flexci/internal/frontend/util/idutil.(*LazyIDGenerator).Do (generator.go:179)
    github.com/pfnet/flexci/internal/frontend/handler/apihandler.(*setJobHandlerFlow).doCreateAction (set_job.go:211)
    github.com/pfnet/flexci/internal/frontend/handler/apihandler.(*setJobHandlerFlow).doAction.func1.1 (set_job.go:173)
    github.com/pfnet/flexci/internal/frontend/handler/apihandler.(*setJobHandlerFlow).doAction.func1 (set_job.go:184)
    github.com/pfnet/flexci/internal/frontend/database.transactionOnce.func1 (database.go:338)
    google.golang.org/appengine/internal.RunTransactionOnce (transaction.go:93)
    google.golang.org/appengine/datastore.RunInTransaction (transaction.go:72)
    github.com/pfnet/flexci/internal/frontend/database.transactionOnce (database.go:336)
    github.com/pfnet/flexci/internal/frontend/database.Transaction (database.go:300)
    github.com/pfnet/flexci/internal/frontend/handler/apihandler.(*setJobHandlerFlow).doAction (set_job.go:169)
    github.com/pfnet/flexci/internal/frontend/handler/apihandler.(*setJobHandlerFlow).Do (set_job.go:57)
    github.com/pfnet/flexci/internal/frontend/handler/apihandler.setJobHandler (set_job.go:31)
    github.com/pfnet/flexci/internal/frontend/handler/apihandler.(*githubWebhookIssueCommentFlow).triggerTest (github_issue_comment.go:289)
    github.com/pfnet/flexci/internal/frontend/handler/apihandler.(*githubWebhookIssueCommentFlow).Do (github_issue_comment.go:100)
    github.com/pfnet/flexci/internal/frontend/handler/apihandler.githubIssueCommentHandler (github_issue_comment.go:48)
    runtime.call64 (asm_amd64.s:523)
    reflect.Value.call (value.go:447)
    reflect.Value.Call (value.go:308)
    github.com/pfnet/flexci/internal/frontend/core.(*registerHandlerFlow).registerHandler.func1.1 (handler.go:184)
    github.com/pfnet/flexci/internal/frontend/core.(*apiHandlerFlow).callHandler (handler.go:472)
    github.com/pfnet/flexci/internal/frontend/core.(*apiHandlerFlow).doInternal (handler.go:324)
    github.com/pfnet/flexci/internal/frontend/core.(*apiHandlerFlow).Do (handler.go:283)
    github.com/pfnet/flexci/internal/frontend/core.(*registerHandlerFlow).registerHandler.func1 (handler.go:181)
    github.com/pfnet/flexci/internal/frontend/core.(*handlerFlow).Do (handler.go:719)
    github.com/pfnet/flexci/internal/frontend/core.(*registerHandlerFlow).Register.func1.1 (handler.go:120)
    github.com/pfnet/flexci/internal/frontend/util/structuredlog.WithContext (log.go:77)
    github.com/pfnet/flexci/internal/frontend/core.(*registerHandlerFlow).Register.func1 (handler.go:116)
    net/http.HandlerFunc.ServeHTTP (server.go:1964)
    net/http.(*ServeMux).ServeHTTP (server.go:2361)
    github.com/pfnet/flexci/internal/common/api.callInternal.func2 (call.go:204)
    github.com/pfnet/flexci/internal/common/api.callInternal (call.go:212)
    github.com/pfnet/flexci/internal/common/api.Call (call.go:128)
    github.com/pfnet/flexci/internal/common/api.GithubIssueComment (call.go:519)
    github.com/pfnet/flexci/internal/frontend/handler/xternalhandler.(*githubHookFlow).doInternal (github_webhook.go:146)
    github.com/pfnet/flexci/internal/frontend/handler/xternalhandler.(*githubHookFlow).Do (github_webhook.go:39)
    github.com/pfnet/flexci/internal/frontend/handler/xternalhandler.githubWebhookHandler (github_webhook.go:29)
    github.com/pfnet/flexci/internal/frontend/core.(*handlerFlow).Do (handler.go:719)
    github.com/pfnet/flexci/internal/frontend/core.(*registerHandlerFlow).Register.func1.1 (handler.go:120)
    github.com/pfnet/flexci/internal/frontend/util/structuredlog.WithContext (log.go:81)
    github.com/pfnet/flexci/internal/frontend/core.(*registerHandlerFlow).Register.func1 (handler.go:116)
    net/http.HandlerFunc.ServeHTTP (server.go:1964)
    net/http.(*ServeMux).ServeHTTP (server.go:2361)
    google.golang.org/appengine/internal.executeRequestSafely (api.go:165)
    google.golang.org/appengine/internal.handleHTTP (api.go:124)
    net/http.HandlerFunc.ServeHTTP (server.go:1964)
    net/http.serverHandler.ServeHTTP (server.go:2741)
    net/http.(*conn).serve (server.go:1847)
    runtime.goexit (asm_amd64.s:1333)

@belltailjp belltailjp changed the title [WIP] Fix _ObjectReader.seek(0, os.SEEK_END) case [Test] Fix _ObjectReader.seek(0, os.SEEK_END) case Dec 22, 2021
@belltailjp
Copy link
Member Author

/test

@pfn-ci-bot
Copy link

Successfully created a job for commit eb422cd:

1 similar comment
@pfn-ci-bot
Copy link

Successfully created a job for commit eb422cd:

@belltailjp belltailjp changed the title [Test] Fix _ObjectReader.seek(0, os.SEEK_END) case [Ready] Fix _ObjectReader.seek(0, os.SEEK_END) case Dec 23, 2021
@kuenishi kuenishi merged commit ddb148d into pfnet:master Dec 23, 2021
@kuenishi kuenishi mentioned this pull request Dec 23, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cat:bug Bug report or fix.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants