feat(storage-service): complete file-listing metadata + fix SFTP file delete#652
Merged
Merged
Conversation
…metadata The gRPC UserStorageService file-listing metadata (list_dir / get_file_metadata) only carried name/path/size/is_directory, so clients lost the modified time and content type the file browser shows. Populate them server-side: - FileMetadata gains modifiedTime (epoch millis) and contentType. - SSHJStorageAdaptor.getFileMetadata sets modifiedTime from the SFTP stat mtime (seconds -> millis) and guesses contentType from the file name. - UserStorageGrpcService.toFileMetadataResponse copies both into FileMetadataResponse (modified_time / content_type), which already declared those proto fields. (created_time is not available over SFTP, so it remains unset.) Validated live against the running server: an uploaded file's get_file_metadata and list_dir now return modified_time (the upload time) and content_type (text/plain for a .txt). storage-service + compute-service unit tests: 35 pass.
yasithdev
changed the title
yasithdev
changed the title
yasithdev
added a commit
to apache/airavata-portals
that referenced
this pull request
Jun 9, 2026
…k D, D4.3) (#190) Repoint the storage path views and the create-user-storage-dir signal from the legacy airavata_django_portal_sdk user_storage helpers to the gRPC storage facade: - UserStoragePathView get/post/put/delete + ExperimentStoragePathView → storage.dir_exists/list_dir/get_file_metadata/create_dir/delete_dir/delete_file (+ _storage_upload_and_register for uploads). New _user_storage_path resolves a request path to the absolute ~/-prefixed path the facade expects, including experiment-relative paths (via the experiment data dir). - New grpc_adapters.user_storage_file / user_storage_directory map the gRPC FileMetadataResponse (name/path/size/modified_time/content_type/data_product_uri, populated server-side by apache/airavata#652) to the dicts the storage serializers read. - UserStorageFileSerializer.downloadURL now builds the portal download-file URL directly from the file's data product URI; UserHasWriteAccessToPathSerializer drops the remote-API user_storage.listdir branch (the gRPC path talks to the backend directly, not a remote portal proxy). - signals.create_user_storage_dir → storage.create_dir/create_symlink. - The deprecated download_file view redirects to the gRPC download-file endpoint. - Drop the now-unused user_storage (and queue_settings_calculators) imports from views.py/serializers.py. REST/JSON contract unchanged. Depends on apache/airavata#652 (per-file modified-time/content-type/data-product-uri in the listing + SFTP file delete). Validated live against the running backend: UserStoragePathView GET lists ~/tmp (200) with each file rendering name/size/createdTime/modifiedTime/mimeType/ dataProductURI/downloadURL/userHasWriteAccess; upload (create dir + register), content overwrite, and DELETE (204, file removed) all round-trip; manage.py check clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Make the gRPC
UserStorageServiceusable as the portal's file backend: fill in the per-file listing metadata the file browser needs, and fix file deletion. Three cohesive parts (all storage-service):1. modified-time + content-type in listings
list_dir/get_file_metadata/list_experiment_dironly carried name/path/size/is_directory. Now:FileMetadatagainsmodifiedTime(epoch millis) andcontentType.SSHJStorageAdaptor.getFileMetadatasetsmodifiedTimefrom the SFTPstatmtime (s→ms) and guessescontentTypefrom the file name. (created_timeisn't available over SFTP, so it stays unset.)UserStorageGrpcService.toFileMetadataResponsecopies both into the response.2. per-file data-product URI
Listings now carry a stable data product URI per file, so clients reference each stored file by data product (downloads, deletes, selection) without a separate path→data-product table.
DataProductInterface/DataProductRepository:getDataProductByReplicaFilePath(JPQL join on the replicafilePath+ gateway).StorageProvider/StorageProviderImpl:getOrCreateDataProductByPathreturns that product's URI, registering a newFILEdata product (singleGATEWAY_DATA_STOREreplica) if none exists.UserStorageGrpcServicepopulatesFileMetadataResponse.data_product_uriper file — best-effort (directories and lookup failures yield an empty URI, never a listing failure).3. fix SFTP file delete
deleteFileranrm -fviaadaptor.executeCommand, but the SFTP adaptor does not support command execution, so every file delete failed with "Command execution not supported on storage resources". AddeddeleteFileto the storage adaptor (SFTPrm) and call it — mirroring howdeleteDiralready usesdeleteDirectory(SFTPrmdir).Together these unblock the Django portal's storage-listing migration off the legacy Thrift SDK and remove the portal's need for its own
UserFilesORM mapping.Validation
get_file_metadata/list_dirreturnmodified_time(upload time),content_type(text/plainfor.txt), and adata_product_urithat is stable across calls (found, not re-registered) and resolves via the data-product service to the file's replica; uploading then deleting a file now succeeds. Full portal CRUD round-trips through the views.mvn test -pl airavata-api/storage-service,airavata-api/compute-serviceand research-service tests pass (0 failures).