refactor: support sourcing images from either file path or in-memory data frame #1174
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Why: renaming variables to support either a string for file path to image or image stored as numpy array in input data set.
Why: Able to accept either file path string or numpy array for image feature.
Why: unit test test_basic_image_feature test sourcing images from both file and as ndarrays in dataframe.
Why: Accounting for possible settings by the user.
Why: Accounting for possible settings by the user.
Why: To bypass test combination that causes returning index values instead of image ndarrays for training
w4nderlust
reviewed
May 7, 2021
w4nderlust
reviewed
May 7, 2021
w4nderlust
reviewed
May 7, 2021
why: to comply with design standard on use of COLUMN key in feature dictionary and added helper function to declutter a long statement.
why: To explain rationale for special handling of unit test where images are sourced from the file system.
why: explict test to ensure the rest api call completed successfully for /predict and /batch_predict endpoints.
why: put in scaffolding to eventually support use of ndarray as image input for test.
why: support new capability int image feature. To support sending ndarray objects in REST api call, added helper functions to ludwig.utils.data_utils.py for serializing/deserializing ndarray objects to/from ludwig custom string format.
Why: Support passing skip_save_processed_input parameter to add_feature_data() methods. This will allow the methods to customize their setup based on the setting of skip_save_processed_input. Immediate need is to support image feature setup to support ndarray support.
|
Ready for review. summary of changes:
|
w4nderlust
reviewed
May 11, 2021
Why: Remove need for custom ludwig string format to handle ndarrays. Made more robust in file handling. Add capability to restore dtype for ndarray are set the same as in the original data source. Create ludwig.utils.server_utils to house the new serialize/deseriallize functions.
why: renamed variables to make more consistent. corrected a few log messages.
|
Summary of changes:
Work remaining:
|
remove code made obsolete by PR ludwig-ai#1174 and address minor todos
provide additional details on how serialize helper function works.
removed code for the deprecated helper functions for ludwig custom string format.
|
Converted back to draft mode. Clarification of requirements is forcing a rethink of current approach. |
Why: replace function _write_file() with _read_image_buffer() this will eliminate need to write temporary files and do clean up of the temporary files. Remove helper functions for custom ludwig string format of ndarray.
Why: rename img_source to img_entry to be consistent with rest of code base.
Why: Use of Dask backend and use of hdf5 cache is incompatibile. With the recent change to support ndarray for images, there are two conditions that now affect if hdf5 cache is used for images. This change moves the backend test for use of hdf5 cache earlier in processing to avoid interactions with the two conditions that are currently in place.
Why: To support audio features that have not been been updated to support an option for ndarray representation. Add unit test for audio feature for model serving.
Why: allow testing of multiple or single record batch as independent tests.
Why: To be addressed as part of long-term update to the audio feature.
…images_from_memory
w4nderlust
approved these changes
May 23, 2021
This was referenced May 23, 2021
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Code Pull Requests
PR addresses Issue #484 and Issue #268.
Adds option to support sourcing images from file path or from in-memory as numpy arrays.