Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

documents: add files support #3639

Merged
merged 9 commits into from
May 15, 2024
Merged

documents: add files support #3639

merged 9 commits into from
May 15, 2024

Conversation

jma
Copy link
Contributor

@jma jma commented Mar 26, 2024

  • Uses rero-invenio-files to add file support to documents.
  • Pushes full text into the document index.
  • Creates a jsonschema store to support local and bib.rero.ch prefix.
  • Adds full text boosting field.
  • Creates randomly files during the setup.
  • Adds documents with files to the 'online' filter.
  • Removes the fulltext fields from the default fields search.
  • Adds a new url params "&fulltext=1" to add the fulltext fields to the search fields.
  • Adds files locations configuration.
  • Closes: Search: exclusion operator doesn't work as expected for most fields #2971.

@jma

This comment was marked as outdated.

Sign in to view
@github-actions github-actions bot added developers From a developer point of view. f: search f: data migration Data migration from a legacy system or a previous version f: permissions Concerns the rights management dev: fixtures Fixtures data used for ils.test and ilsdev.test f: activity-logs Everything around logging user or system activities labels Mar 26, 2024
@github-actions github-actions bot added the f: statistics Related to the usage statistics either for pricing or for the libaries reports label Apr 11, 2024
@PascalRepond

This comment was marked as outdated.

Sign in to view
@mmo

This comment was marked as outdated.

Sign in to view
@github-actions github-actions bot added the f: data About data model, importation, transformation, exportation of data, specific for bibliographic data label May 2, 2024
@PascalRepond
Copy link
Contributor

PascalRepond commented May 8, 2024 •
edited by jma

Tests 08.05

  • Also update the librarian and legacy stats (but not pricing) for "number of documents" to count documents with only files (use document.organisation_pid and library_pid) instead of "holdings" same as for the facets.
    • Specs: "Librarian stats config (+ legacy librarian stats): the number_of_documents indicator count files as belonging to a library if they have files in this library (even if they have no holdings in this library) == facets"
  • Pro UI: the list of files is not displayed in the "Files" tab
  • Pro UI: in the list of files in the "Files" tab, the link to download a file (title + icon) should open in a new tab (same as for the "get" list)
  • REP/13.05: Console error when trying to save a digital collection and collection is not saved

@jma jma force-pushed the US2172-files branch 2 times, most recently from 5933e22 to 3aa332e Compare May 14, 2024 11:32
jma added 9 commits May 15, 2024 08:06
@jma
documents: add files support
165e724 
- Uses rero-invenio-files to add file support to documents.
- Pushes full text into the document index.
- Creates a jsonschema store to support local and bib.rero.ch prefix.
- Adds full text boosting field.
- Creates randomly files during the setup.
- Adds documents with files to the 'online' filter.
- Removes the fulltext fields from the default fields search.
- Adds a new url params "&fulltext=1" to add the fulltext fields to the search fields.
- Adds files locations configuration.
- Closes: #2971.

Co-Authored-by: Johnny Mariéthoz <Johnny.Mariethoz@rero.ch>
@jma
files: add file permissions configurations
0c9947c 
* Adds record files service permission configurations.
* Adds invenio record files support for the permission web API.
* Avoids document deletion if files exists.
* Changes the max number of files per bucket.
* Limits the upload size to 500Mb by default.

Co-Authored-by: Johnny Mariéthoz <Johnny.Mariethoz@rero.ch>
@jma
files: reindex the linked document
d0872ef 
* Reindexes the linked document when a file is modified in the related record file.
* Creates a cli to attach files to a documents for a given library.
* Adds missing configurations for tasks.
* Uses `Unit Of Work` invenio concept to reindex the document only once even if
  multiple files are added using the cli.

Co-Authored-by: Johnny Mariéthoz <Johnny.Mariethoz@rero.ch>
@jma
files: add operation logs for files CRUD operations.
12baf8a 
* Uses invenio records resources components to create operation logs.

Co-Authored-by: Johnny Mariéthoz <Johnny.Mariethoz@rero.ch>
@jma
files: inject files information in the record index.
a456216 
* Adds custom mappings and jsonschema.
* Reindex record file when a files is added, updated or removed.
* Adds the total size taken by the files and the number of main files related to a given record file.

Co-Authored-by: Johnny Mariéthoz <Johnny.Mariethoz@rero.ch>
@jma
files: add statistics for pricing
b16f863 
* Injects the file library pid in to the document.
* Adds number_of_docs_with_files, number_of_files, files_volume princing stats.

Co-Authored-by: Johnny Mariéthoz <Johnny.Mariethoz@rero.ch>
@jma
files: add file preview
4efe88f 
* Adds and configures invenio-previewer.
* Increases the body size for files url in the nginx configuration.
* Removes useless code.
* Fixes reindex the related document when a record file collections has been changed.
* Removes thumbnails and fullext from the REST files list.
* Changes the maximum file limit.

Co-Authored-by: Johnny Mariéthoz <Johnny.Mariethoz@rero.ch>
@jma
files: enable organisation aggregations
0c3aaf3 
* Creates a new organisation, library index field to aggregate the files and holdings organisation informations.
* Injects the organisation files informations in the document index.
* Fixes some export format and entities organisation based document query.
* Adds organisation informations in the record file index.
* Adapts the public view viewcode filtering.

Co-Authored-by: Johnny Mariéthoz <Johnny.Mariethoz@rero.ch>
@jma
files: uses references for library and document
09030f8 
* Replaces owners and links to library and document $ref in the record file.
* Reverts uwsgi files modified by mistake.
* Fixes spelling in some function names.

Co-Authored-by: Johnny Mariéthoz <Johnny.Mariethoz@rero.ch>
@jma jma marked this pull request as ready for review May 15, 2024 06:18
@jma jma merged commit 6ff907d into staging May 15, 2024
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
dev: fixtures Fixtures data used for ils.test and ilsdev.test developers From a developer point of view. f: activity-logs Everything around logging user or system activities f: data migration Data migration from a legacy system or a previous version f: data About data model, importation, transformation, exportation of data, specific for bibliographic data f: permissions Concerns the rights management f: search f: statistics Related to the usage statistics either for pricing or for the libaries reports
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Search: exclusion operator doesn't work as expected for most fields
3 participants
@jma @PascalRepond @mmo