Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: File #16289

Merged
merged 59 commits into from
Jun 8, 2022
Merged

refactor: File #16289

merged 59 commits into from
Jun 8, 2022

Conversation

gavindsouza
Copy link
Collaborator

@gavindsouza gavindsouza commented Mar 15, 2022

Changes

  • Restructured File module

./frappe/core/doctype/file/file.py -> [./frappe/core/api/file.py, ./frappe/core/doctype/file/utils.py, ./frappe/core/doctype/file/exceptions.py, ./frappe/core/doctype/file/file.py]

  • Maintain import paths & API endpoints
  • New namespace for file based endpoints actions -> /api/method/frappe.core.api.file
  • Simplify interfaces for File actions
  • Added type hints to aid my development & debugging
  • Bug & security fixes

Fixes

  • Disallow file access beyond the respective site's folder
  • Consistent setting of File.file_name attribute
  • Avoid overwriting files that may have the same name and suffix
  • Enforce the "max_number_of_files" restriction in FileUploader
  • Make File tests atomic
  • Improved syncing file system with changes on rollback
  • Cover extension-less img previews in File form

Other Changes

  • Skip test_multiple_doctypes_sync for Postgres test run

@codecov
Copy link

codecov bot commented Mar 15, 2022

Codecov Report

Merging #16289 (44dba28) into develop (b418aea) will decrease coverage by 5.37%.
The diff coverage is 81.42%.

@@             Coverage Diff             @@
##           develop   #16289      +/-   ##
===========================================
- Coverage    58.49%   53.11%   -5.38%     
===========================================
  Files          772      758      -14     
  Lines        69190    67837    -1353     
  Branches      6012     5749     -263     
===========================================
- Hits         40471    36030    -4441     
- Misses       25091    27736    +2645     
- Partials      3628     4071     +443     
Flag Coverage Δ
server 61.23% <81.57%> (-2.76%) ⬇️
ui-tests 38.22% <0.00%> (-10.79%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

* Separate out utils, exceptions & core / maintain import paths [except
  API endpoints]
* Add type hints
* Simplify logic
Other fix:
* Don't set File.file_name (too) different from File.file_url
Restructured and moved most APIs under frappe.core.api.file namespace.
Changed some obvious security gaps (like using get_list instead of
get_all for an endpoint), styled, added type hints and made minor performance
enhancements.

Changes
* download_file API
    * Move API to handler.py
    * Check for permissions via File.is_downloadable instead
* Moved APIs to new namespace: `frappe.core.api.file`
* Backwards compatibility
    * Added APIs to override_whitelisted_methods to maintain existing
      client endpoints
    * Imported APIs to controller's namespace to avoid breaking external
      app usages
The old paths worked too, but it's just better to use the new paths and
not go in circles ;)
* Simplify object lifecycle flows
* Re-organize and separate distinct actions / APIs
* Style black-ish
* Simplify logic in methods "touched"
@rmehta
Copy link
Member

rmehta commented Mar 21, 2022

@gavindsouza since you are cleaning up file, please also fix the bug where if you select a file as a property in a document, it creates a new file record. (example: attach a file to a blog, and then select it as cover - 2 files records get created)!

Had discussed this with @surajshetty3416

* Generalize file content reverting on rollback
* Overwrite current file instead of adding a dangling new file
* Use File.save_file instead of additional "custom" logic
* Set File.content on calling get_content method
* Make file renaming atomic
* Add rollback observer to maintain filesystem consistency
* Simplify logic
* Set default Falsy value for content
* Extend Frappe exceptions into file's
Delete, Move or Restore files in the filesystem by setting Document
flags in the on_rollback method
* Move checks into common methods
* Refactor to use latest APIs
* Remove seemingly unnecessary code
* Fix previously undhanled use cases
@gavindsouza gavindsouza linked an issue Apr 26, 2022 that may be closed by this pull request
@gavindsouza
Copy link
Collaborator Author

Screenshot 2022-04-26 at 5 02 45 PM

Fixed #13624 by ^

@gavindsouza gavindsouza marked this pull request as ready for review April 27, 2022 05:39
@gavindsouza gavindsouza requested a review from a team April 27, 2022 05:39
@stale stale bot added the inactive label May 16, 2022
@frappe frappe deleted a comment from stale bot May 16, 2022
@stale stale bot removed the inactive label May 16, 2022
@stale stale bot added the inactive label May 24, 2022
@frappe frappe deleted a comment from stale bot May 24, 2022
@stale stale bot removed the inactive label May 24, 2022
@gavindsouza
Copy link
Collaborator Author

Shall muster the courage to revisit soon 🥲

@stale
Copy link

stale bot commented May 31, 2022

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed within 3 days if no further activity occurs, but it only takes a comment to keep a contribution alive :) Also, even if it is closed, you can always reopen the PR when you're ready. Thank you for contributing.

@stale stale bot added the inactive label May 31, 2022
@surajshetty3416 surajshetty3416 marked this pull request as draft June 1, 2022 08:23
@stale stale bot removed the inactive label Jun 1, 2022
it knows

* Hide Preview if request is errored. EAFP.
* Set file attachment limit based on how many attachments have already
  been added. If 1 attachment exists and the limit is 3, File uploader
will allow you to select only 2 more files.
@gavindsouza gavindsouza marked this pull request as ready for review June 8, 2022 07:13
@gavindsouza
Copy link
Collaborator Author

Ignoring flaky Cypress test 'Verifying data control by inputting different patterns for "Email" field' (cc: @Komal-Saraf0609)

@gavindsouza gavindsouza merged commit 67736ef into frappe:develop Jun 8, 2022
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 16, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Attachments overwrites files with same name File Attachment Limit not being set properly
3 participants