Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crypto/sys_fs: NPDRM decryption performance improvememts, memory usage reduction #15139

Merged
merged 4 commits into from
Feb 5, 2024

Conversation

elad335
Copy link
Contributor

@elad335 elad335 commented Feb 2, 2024

  • Previous PKG installer code was structured very naively: if 8GB read is requested, it would allocate 8GB of memory. Without considering recycling memory for each block.

TODO:

  • Cache system for NPDRM reads in order to make PS3 filesystem reads much faster.
  • Avoid reading entire data blocks if only few bytes are requested in decrypt_block, this should only apply when compression is not used inisde of the EDAT blocks.

Additional / Side effects:

  • EDATADecrypter::ReadData is now thread-safe (due to use of local buffers instead of global, file::read_at instead of file::read). This fixes a race condition when using the syscall sys_fs_fcntl in block read mode.

rpcs3/Crypto/unedat.cpp Outdated Show resolved Hide resolved
rpcs3/Crypto/unedat.cpp Outdated Show resolved Hide resolved
@elad335
Copy link
Contributor Author

elad335 commented Feb 3, 2024

Found another and more severe performance issue: when installing SDATs from PKG files data would first be decrypted as a mid-stage to disk, then loaded in whole to RAM, then flushed to destination as final decrypted data.
Instead of this atrocity it reads data now as needed and flushes final data as it reads the encrypted data, without writing out to disk mid-stage data nor keeping insane amounts in RAM.

@elad335 elad335 force-pushed the npdrm branch 7 times, most recently from 3b60965 to 2b3febf Compare February 4, 2024 09:29
@elad335 elad335 marked this pull request as ready for review February 4, 2024 09:29
u64 size_left = edat->file_size;
std::unique_ptr<u8[]> data(new u8[edat->block_size]);

std::vector<u8> data(edat->block_size);

for (int i = 0; i < total_blocks; i++)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

signed mismatch

@@ -326,22 +320,24 @@ s64 decrypt_block(const fs::file* in, u8* out, EDAT_HEADER *edat, NPD_HEADER *np
// reset file to beginning of data before calling
int decrypt_data(const fs::file* in, const fs::file* out, EDAT_HEADER *edat, NPD_HEADER *npd, unsigned char* crypt_key, bool /*verbose*/)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should return a bool

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed

@elad335 elad335 force-pushed the npdrm branch 8 times, most recently from ee01ac3 to 6dacf3d Compare February 4, 2024 20:43
@elad335 elad335 changed the title Crypto/sys_fs: Fix potential RAM shortage in NPDRM decryption Crypto/sys_fs: NPDRM decryption performance improvememts, memory usagee reduction, cache system initial implementation Feb 4, 2024
@elad335 elad335 added the Optimization Optimizes existing code label Feb 4, 2024
@elad335 elad335 force-pushed the npdrm branch 2 times, most recently from bd17764 to 307b184 Compare February 5, 2024 11:56
Previous code was structured very naively: if 8GB read is requested, it would allocate 8GB of memory. Without considering recycling memory for each block.
@elad335 elad335 merged commit 40c51a1 into RPCS3:master Feb 5, 2024
6 checks passed
@elad335 elad335 changed the title Crypto/sys_fs: NPDRM decryption performance improvememts, memory usagee reduction, cache system initial implementation Crypto/sys_fs: NPDRM decryption performance improvememts, memory usage reduction Feb 5, 2024
@elad335 elad335 deleted the npdrm branch May 6, 2024 15:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants