Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added support for mounting file storage for oci job #139

Merged
merged 15 commits into from
Apr 29, 2023

Conversation

lu-ohai
Copy link
Member

@lu-ohai lu-ohai commented Apr 7, 2023

Added support for mounting file storage for oci job

  • Added new API with_storage_mount() and storage_mount for DataScienceJob:
  • Added new dataclass DSCFileStorage to hold the required arguments for file system mounting
.with_storage_mount(
    {
        "src":"100.100.86.140:/FileSystem-20230330-2239-01",
        "dest":"test_mount",
    }
)

Unit test
Screen Shot 2023-04-07 at 11 15 47 AM

@oracle-contributor-agreement oracle-contributor-agreement bot added the OCA Verified All contributors have signed the Oracle Contributor Agreement. label Apr 7, 2023
@lu-ohai lu-ohai requested review from mrDzurb and qiuosier April 7, 2023 18:17
@qiuosier qiuosier requested review from mayoor and z7ye April 7, 2023 18:28

Parameters
----------
storage_mount : List[Union(DSCFileSystem, dict)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the format of the dict?

Could you please add code example here?

Copy link
Member Author

@lu-ohai lu-ohai Apr 13, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The format would be something as this

.with_storage_mount(
    {
        storage_type="FILE_STORAGE", 
        destination_directory_name="test_folder_name", 
        export_path="test_export_path", 
        mount_target="test_mount_target"
    },
    {
        storage_type="OBJECT_STORAGE", 
        destination_directory_name="test_folder_name", 
        namespace="test_namespace", 
        path="test_path"
    }, 
)

Two keys are required for each dict: storage_type and destination_directory_name
The other keys need to be provided based on which type of file systems users would like to mount. For file storage, these are export_path and mount_target. For object storage, these are namespace and path.
We'll have a pre-check for the keys for each type passed in so that user will be notified immediately if anything is missing or wrong.

ads/jobs/__init__.py Outdated Show resolved Hide resolved
ads/jobs/builders/infrastructure/dsc_file_system.py Outdated Show resolved Hide resolved
ads/jobs/builders/infrastructure/dsc_file_system.py Outdated Show resolved Hide resolved
ads/jobs/builders/infrastructure/__init__.py Outdated Show resolved Hide resolved
@lu-ohai lu-ohai marked this pull request as draft April 13, 2023 19:54
ads/jobs/__init__.py Outdated Show resolved Hide resolved
ads/jobs/builders/infrastructure/__init__.py Outdated Show resolved Hide resolved
ads/jobs/builders/infrastructure/dsc_file_system.py Outdated Show resolved Hide resolved
ads/jobs/builders/infrastructure/__init__.py Outdated Show resolved Hide resolved
ads/jobs/builders/infrastructure/dsc_job.py Outdated Show resolved Hide resolved
ads/jobs/builders/infrastructure/dsc_file_system.py Outdated Show resolved Hide resolved
ads/jobs/builders/infrastructure/dsc_job.py Outdated Show resolved Hide resolved
ads/jobs/builders/infrastructure/dsc_file_system.py Outdated Show resolved Hide resolved
ads/jobs/builders/infrastructure/dsc_file_system.py Outdated Show resolved Hide resolved
@lu-ohai lu-ohai requested review from qiuosier and mayoor April 26, 2023 22:25
qiuosier
qiuosier previously approved these changes Apr 28, 2023
ads/jobs/builders/infrastructure/dsc_job.py Outdated Show resolved Hide resolved
@lu-ohai lu-ohai marked this pull request as ready for review April 29, 2023 00:06
@lu-ohai lu-ohai merged commit 3a38dcf into oracle:develop Apr 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
OCA Verified All contributors have signed the Oracle Contributor Agreement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants