Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

11948 Use pre-existing local file for import #3996

Merged
merged 60 commits into from
Aug 24, 2023
Merged

Conversation

paul-m
Copy link
Contributor

@paul-m paul-m commented Aug 1, 2023

  • Since it was unclear exactly which layer should contain this solution, I ended up reading a lot of code and thus doing minor refactoring in some places, and adding a lot of docblocks and CS fixes for some files.
    • Notably, DataResource now has getUniqueIdentifierNoPerspective(), to go along with its existing getUniqueIdentifier(), and updating places where the no perspective id was being created on the fly.
    • This all works towards making the whole thing more readable overall.
  • Adds a new DKAN common config called always_use_existing_local_perspective. It’s a bool, and makes your DKAN override the file fetcher to use an existing file rather than copy one from the source.
  • Adds a warning to the site status page telling you that you have always_use_existing_local_perspective enabled.
  • \Drupal\common\FileFetcher\Factory is now \Drupal\common\FileFetcher\FileFetcherFactory. This name change comes along with the ability to check the always_use_existing_local_perspective configuration and use our new file-fetcher remote processor depending on what it says.
  • Adds \Drupal\common\FileFetcher\FileFetcherRemoteUseExisting, which is a file fetcher processor to replace Remote if we want to use the existing file.
  • Adds tests to prove all this stuff does what it says.
  • Added a page of documentation: https://dkan--3996.org.readthedocs.build/en/3996/user-guide/guide_local_files.html

There's no integrity check performed on the file, other than whether it exists in the file system... For instance the file could exist but be zero bytes long. We'll follow up with issues about this process.

@paul-m paul-m marked this pull request as ready for review August 10, 2023 17:29
Copy link
Member

@dafeder dafeder left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing work here @paul-m. Take a look at a few suggestions and questions I had.

docs/source/user-guide/guide_local_files.rst Outdated Show resolved Hide resolved
docs/source/user-guide/guide_local_files.rst Outdated Show resolved Hide resolved
docs/source/user-guide/guide_local_files.rst Outdated Show resolved Hide resolved
docs/source/user-guide/guide_local_files.rst Outdated Show resolved Hide resolved
docs/source/user-guide/guide_local_files.rst Outdated Show resolved Hide resolved
docs/source/user-guide/guide_local_files.rst Outdated Show resolved Hide resolved
modules/common/common.install Outdated Show resolved Hide resolved
modules/common/common.install Show resolved Hide resolved
modules/common/src/DataResource.php Show resolved Hide resolved
modules/datastore/src/Drush.php Show resolved Hide resolved
paul-m and others added 15 commits August 22, 2023 13:52
Co-authored-by: Dan Feder <dan.feder@civicactions.com>
Co-authored-by: Dan Feder <dan.feder@civicactions.com>
Co-authored-by: Dan Feder <dan.feder@civicactions.com>
Co-authored-by: Dan Feder <dan.feder@civicactions.com>
Co-authored-by: Dan Feder <dan.feder@civicactions.com>
Co-authored-by: Dan Feder <dan.feder@civicactions.com>
Co-authored-by: Dan Feder <dan.feder@civicactions.com>
Co-authored-by: Dan Feder <dan.feder@civicactions.com>
Co-authored-by: Dan Feder <dan.feder@civicactions.com>
Co-authored-by: Dan Feder <dan.feder@civicactions.com>
Co-authored-by: Dan Feder <dan.feder@civicactions.com>
Co-authored-by: Dan Feder <dan.feder@civicactions.com>
Copy link
Member

@dafeder dafeder left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved, looks like we need to re-run a cypress test though

@dafeder dafeder merged commit 2679fab into 2.x Aug 24, 2023
12 checks passed
@dafeder dafeder deleted the 11948-local-filefetcher branch August 24, 2023 18:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants