Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make data file names POSIX compliant #2732

Merged
merged 1 commit into from Apr 27, 2018
Merged

Conversation

hackdna
Copy link
Member

@hackdna hackdna commented Apr 25, 2018

Fix #2553

@codecov
Copy link

codecov bot commented Apr 25, 2018

Codecov Report

Merging #2732 into develop will decrease coverage by 0.25%.
The diff coverage is 100%.

Impacted file tree graph

@@             Coverage Diff             @@
##           develop    #2732      +/-   ##
===========================================
- Coverage    56.11%   55.86%   -0.26%     
===========================================
  Files          398      398              
  Lines        25583    25423     -160     
  Branches      1240     1239       -1     
===========================================
- Hits         14356    14202     -154     
+ Misses       11227    11221       -6
Impacted Files Coverage Δ
refinery/file_store/test_models.py 100% <100%> (ø) ⬆️
refinery/file_store/models.py 72.72% <100%> (-1.56%) ⬇️
refinery/tool_manager/tasks.py 100% <0%> (ø) ⬆️
refinery/tool_manager/views.py 96.22% <0%> (+1.32%) ⬆️
...s/data-set-visualization/services/visualization.js 94.73% <0%> (+7.23%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 81c5878...347be7f. Read the comment docs.

@@ -47,15 +48,11 @@ def _mkdir(path):

def file_path(instance, filename):
"""Construct relative file system path for new file store files relative to
FILE_STORE_BASE_DIR.
FILE_STORE_BASE_DIR
instance: FileStoreItem instance
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't look like instance ever gets used here

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct. However, file_path is passed to upload_to as a callable.
"This callable must be able to accept two arguments"
https://docs.djangoproject.com/en/1.8/ref/models/fields/#django.db.models.FileField.upload_to

@@ -84,6 +75,43 @@ def test_file_source_map(self):
'/example/path/test_file.dat')


class FilePathTest(TestCase):

def test_file_path_format(self):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't believe file_path needs a FileStoreItem instance in any of these tests

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The signature of file_path requires a FileStoreItem instance.

# max length: 255, allowed character set: [A-Za-z0-9._-]
# remove leading '-' characters to make file management easier
filename = re.sub('[^A-Za-z0-9._-]', '_',
string.lstrip(filename, '-'))[-255:]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the idea that the last part of the filename has the extension, and you definitely want to preserve that? Hopefully we never get files this long, but for people, the start of the name is also significant... it'd be more complicated, and probably not worth it, but I could imagine removing the part of the name just before the extension.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think most file systems in use today have a limit of 255 on file name length which means there should really be no files that are above that. So this is more of a last line of defense for which simple handling is sufficient.

@hackdna hackdna merged commit 8e16f28 into develop Apr 27, 2018
@hackdna hackdna deleted the hackdna/file-path-fix branch April 27, 2018 16:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants