Skip to content

Fix/prevent same data uploads#262

Merged
joaorafaelalmeida merged 3 commits intoEHDEN:devfrom
BMDSoftware:fix/prevent_same_data_uploads
Jan 5, 2023
Merged

Fix/prevent same data uploads#262
joaorafaelalmeida merged 3 commits intoEHDEN:devfrom
BMDSoftware:fix/prevent_same_data_uploads

Conversation

@joselfrias
Copy link
Copy Markdown
Contributor

Adds hash mechanism to prevent the user or database owner to upload the same data or the same file, equal to the one already in the database, in the dashboards uploader.

Description

This PR adds a new check mechanism when a file is uploaded, namely by verifying if the data or file uploaded
is equal to the stored data for the data source, by evaluating the hashed contents of the uploaded file and
the file previous uploaded to the database, if it exists.
This new step was implemented to act before every other check, such as the metadata ones, which allows
that if the same file is uploaded, the processing can immediately terminate, with no need to
process the other checks, as the upload process will fail.
For now the message presented in the uploader page in the case of a duplicated file is "File is already
present in the database".

Related Issue

Fixes #260

Motivation and Context

The upload of files into the Network Dashboards can become computationally intensive process, in the case of large data, especially due to the refresh of the materialized views that occurs after each upload to incorporate the new data
in the graphic representations. By preventing the dashboard owners to upload the same data and the same files,
we are potentially saving server resources.

How Has This Been Tested?

Unit Tests:

  • a first valid file is uploaded, followed by another upload with the same content, evaluating if the process of upload failed.
  • a first valid file is uploaded, followed by another upload with different content, evaluating if there was no errors in the upload
    process.

Testing Environment:

  • Setup of the docker-compose stack for tests (postgres and redis) with test dependencies (as defined here)

Affected areas of the code:

  • The code added affects the areas related to the upload functionalities, as it is now the first check.

@joaorafaelalmeida joaorafaelalmeida merged commit 17e9c39 into EHDEN:dev Jan 5, 2023
joaorafaelalmeida added a commit that referenced this pull request Jan 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants