Skip to content

Add RemoteFileAccessProvider - #1649

Closed
pingsutw wants to merge 8 commits into
masterfrom
remote-file-access
Closed

Add RemoteFileAccessProvider#1649
pingsutw wants to merge 8 commits into
masterfrom
remote-file-access

Conversation

@pingsutw

Copy link
Copy Markdown
Member

TL;DR

Currently, flytekit remote will try to use s3fs to upload data, but it should use data proxy instead.
RemoteFileAccessProvider allows flytekit remote to upload the data (structure dataset, flyte file, etc) to the s3.

Type

  • Bug Fix
  • Feature
  • Plugin

Are all requirements met?

  • Code completed
  • Smoke tested
  • Unit tests added
  • Code documentation added
  • Any pending items have an associated Issue

Complete description

^^^

Tracking Issue

https://flyte-org.slack.com/archives/CP2HDHKE1/p1684272325647759

Follow-up issue

NA

Signed-off-by: Kevin Su <pingsutw@apache.org>
@codecov

codecov Bot commented May 18, 2023

Copy link
Copy Markdown

Codecov Report

Merging #1649 (29b756c) into master (c8433ea) will increase coverage by 0.02%.
The diff coverage is 74.60%.

❗ Current head 29b756c differs from pull request most recent head d5baabc. Consider uploading reports for the commit d5baabc to get more accurate results

@@            Coverage Diff             @@
##           master    #1649      +/-   ##
==========================================
+ Coverage   71.00%   71.02%   +0.02%     
==========================================
  Files         336      336              
  Lines       30781    30781              
  Branches     5576     5575       -1     
==========================================
+ Hits        21855    21862       +7     
+ Misses       8379     8372       -7     
  Partials      547      547              
Impacted Files Coverage Δ
flytekit/clients/friendly.py 4.89% <ø> (ø)
flytekit/types/schema/types.py 77.60% <0.00%> (ø)
flytekit/core/data_persistence.py 53.60% <63.41%> (+2.50%) ⬆️
flytekit/extras/tensorflow/model.py 85.71% <100.00%> (ø)
flytekit/extras/tensorflow/record.py 88.50% <100.00%> (ø)
flytekit/remote/remote.py 39.42% <100.00%> (+0.15%) ⬆️
tests/flytekit/unit/core/test_data_persistence.py 100.00% <100.00%> (ø)

... and 17 files with indirect coverage changes

pingsutw added 6 commits May 18, 2023 14:03
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
@pingsutw
pingsutw marked this pull request as draft May 24, 2023 21:49
Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
@pingsutw pingsutw closed this Dec 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants