Removing indent in string stored in hdf5 #1228

momchil-flex · 2023-10-31T23:30:20Z

No description provided.

tylerflex · 2023-11-01T14:02:54Z

tidy3d/components/base.py

@@ -563,7 +563,7 @@ def to_hdf5(self, fname: str, custom_encoders: List[Callable] = None) -> None:

        with h5py.File(fname, "w") as f_handle:

-            f_handle[JSON_TAG] = self._json_string
+            f_handle[JSON_TAG] = self._json()


does this indent=None by default? maybe it's more clear to pass it explicitly?

momchil-flex · 2023-11-02T03:31:45Z

I've updated this PR to use indent=None internally, and I've also introduced the splitting of the string in the hdf5 file. The one thing I'm not sure is what to do with web/core, where I find separate functions for reading the string from the hdf5:

tidy3d/tidy3d/web/core/task_core.py

Lines 29 to 34 in 07a27f2

    
           def _read_simulation_from_hdf5(file_name: str): 
        
               """read simulation str from hdf5""" 
        
               with h5py.File(file_name, "r") as f_handle: 
        
                   json_string = f_handle[JSON_TAG][()] 
        
                   return json_string

, and there's one more instance.

I think the logic is that I shouldn't be importing anything from tidy3d/components here, so I can't use something like Tidy3dBaseModel._json_string_from_hdf5? Then options seem to be:

Repeat the code
Introduce a function in something like tidy3d/utils.py that gets imported both in components/base.py and in web/core.py. But does that also go against the future possibility of completely separating web/core.py?

tylerflex

string splitting Looks good, a few suggestions for making it a bit simpler.

tylerflex · 2023-11-02T13:50:38Z

tidy3d/components/base.py

-            f_handle[JSON_TAG] = self._json_string
+            json_str = self._json_string
+            for ind in range(ceil(len(json_str) / MAX_STRING_LENGTH)):
+                key = f"{JSON_TAG}_{ind}" if ind > 0 else JSON_TAG


Maybe we can write a static method or module level function for this to avoid duplicating the key as a function of index, incase we change this or need to use it elsewhere.

in Tidy3DBaseModel

@staticmethod def _json_string_key(index): if index: return f"{JSON_TAG}_{ind}" return JSON_TAG

to_hdf5

json_str = self._json_string num_strings = ceil(len(json_str) / MAX_STRING_LENGTH) for ind in range(num_strings): ind_start = int(ind * MAX_STRING_LENGTH) ind_stop = min(int(ind + 1) * MAX_STRING_LENGTH, len(json_str)) key = self. _json_string_key(ind) f_handle[key] = json_str[ind_start:ind_stop]

@classmethod def _json_string_from_hdf5(cls, fname: str) -> str: """Load the model json string from an hdf5 file.""" with h5py.File(fname, "r") as f_handle: num_string_parts = len([key for key in f_handle.keys() if JSON_TAG in key]) json_string = "" for ind in range(num_string_parts): key = _json_string_key(ind) json_string += f_handle[key][()] return json_string

tylerflex · 2023-11-02T13:51:56Z

tidy3d/components/base.py

-            f_handle[JSON_TAG] = self._json_string
+            json_str = self._json_string
+            for ind in range(ceil(len(json_str) / MAX_STRING_LENGTH)):
+                key = f"{JSON_TAG}_{ind}" if ind > 0 else JSON_TAG


So this creates JSON_STRING, JSON_STRING_1, ...?
I'm curious if it will make the logic simpler to just not perform any special handling for ind=0, like JSON_STRING_0, JSON_STRING_1, .... If the string is short enough to fit in JSON_STRING_0, we just leave it named like that. Will it break backwards compatibility with script loading maybe?

anyway, see comment and suggestion above, I think it makes things a bit simpler to keep the handling how it is but just have a single function to get the key, treating ind=0 differently.

Sounds good, yeah not appending ind=0 is for backwards compatibility.

tests/test_components/test_IO.py

tylerflex · 2023-11-02T14:02:11Z

Regarding web/core, that's MC-controlled and I think we're duplicating code in there until we can unify it at a later date (@dbochkov-flexcompute?) so I'd say maybe we can just duplicate but add a #TODO: to unify with this method later.

dbochkov-flexcompute · 2023-11-02T15:19:19Z

I've updated this PR to use indent=None internally, and I've also introduced the splitting of the string in the hdf5 file. The one thing I'm not sure is what to do with web/core, where I find separate functions for reading the string from the hdf5:

tidy3d/tidy3d/web/core/task_core.py

Lines 29 to 34 in 07a27f2

def _read_simulation_from_hdf5(file_name: str):

"""read simulation str from hdf5"""

with h5py.File(file_name, "r") as f_handle:

json_string = f_handle[JSON_TAG][()]

return json_string

, and there's one more instance.
I think the logic is that I shouldn't be importing anything from tidy3d/components here, so I can't use something like Tidy3dBaseModel._json_string_from_hdf5? Then options seem to be:

Repeat the code

Introduce a function in something like tidy3d/utils.py that gets imported both in components/base.py and in web/core.py. But does that also go against the future possibility of completely separating web/core.py?

Regarding web/core, that's MC-controlled and I think we're duplicating code in there until we can unify it at a later date (@dbochkov-flexcompute?) so I'd say maybe we can just duplicate but add a #TODO: to unify with this method later.

So far, the way we have been dealing with importing tidy3d things into web api is through passing those things during initialization of web api here

tidy3d/tidy3d/web/__init__.py

Line 32 in 07a27f2

core_config.set_config(log, get_logging_console(), __version__)

For example, currently we pass log, logging console (should change that to logging console getter eventually), and tidy3d version. So, perhaps _json_string_from_hdf5() could also be set in web api from tidy3d in the same way? That is:

core_config.set_config(log, get_logging_console(), __version__, Tidy3dBaseModel._json_string_from_hdf5)

tylerflex · 2023-11-02T16:26:22Z

yea perhaps this is a way to do it, although it seems a bit strange potentially.

Maybe an alternative approach is to handle this in the SimulationTask, grab the class of the component for the task and then call cls._json_string_from_hdf5? Just not sure how to grab the class without downloading the whole object.

dbochkov-flexcompute · 2023-11-02T16:37:13Z

yea perhaps this is a way to do it, although it seems a bit strange potentially.

Maybe an alternative approach is to handle this in the SimulationTask, grab the class of the component for the task and then call cls._json_string_from_hdf5? Just not sure how to grab the class without downloading the whole object.

Yeah, I think it would work there. However, the other place where json string is extracted from hdf5 is here

tidy3d/tidy3d/web/api/tidy3d_stub.py

Lines 152 to 169 in 07a27f2

    
           extension = _get_valid_extension(file_path) 
        
           if extension == ".json": 
        
               json_str = read_simulation_from_json(file_path) 
        
           elif extension == ".hdf5": 
        
               json_str = read_simulation_from_hdf5(file_path) 
        
           elif extension == ".hdf5.gz": 
        
               json_str = read_simulation_from_hdf5_gz(file_path) 
        
           data = json.loads(json_str) 
        
           type_ = data["type"] 
        
           if "SimulationData" == type_: 
        
               sim_data = SimulationData.from_file(file_path) 
        
           elif "ModeSolverData" == type_: 
        
               sim_data = ModeSolverData.from_file(file_path) 
        
           elif "HeatSimulationData" == type_: 
        
               sim_data = HeatSimulationData.from_file(file_path) 
        
           return sim_data

and the purpose there is to actually figure our what simulation class it is

tylerflex · 2023-11-02T16:40:26Z

I see, that's a bit messy, maybe the easiest approach is to just duplicate but make a note and get qingeng to help unify things, but passing in the config also works (even though it seems like not the long term solution to me).

momchil-flex · 2023-11-02T21:41:19Z

Updated, with code duplicated in web.core.py.

tylerflex · 2023-11-03T17:28:16Z

tidy3d/components/base.py

-# default indentation (# spaces) in files
-INDENT = 4
+
+INDENT_JSON_FILE = 4  # default indentation of json string in json files


we could consider setting this to 2 to reduce the size of the file itself, but no preference either way.

note: I'm not sure this will actually change the file size. depends on how they json handles indents I guess.

Tested. It does change the size. I guess let's do 2 yeah? But this shouldn't matter for upload/download from the server because all this is done through hdf5 files now. So not as important potentially, could be left 4 since it's prettier?

fine by me. I prefer 4 if the file size is not an issue.

tylerflex

looks good, thank you! just a couple minor notes

tylerflex · 2023-11-03T17:29:13Z

tidy3d/components/base.py

@@ -430,6 +434,23 @@ def get_sub_model(cls, group_path: str, model_dict: dict | list) -> dict:
                    model_dict = model_dict[key]
        return model_dict

+    @staticmethod
+    def _json_string_key(index):


maybe add some type annotations just while we're at this

def _json_string_key(index: int) -> str:

momchil-flex force-pushed the momchil/hdf5_string_indent branch 2 times, most recently from b2ba1cb to 413f0c9 Compare October 31, 2023 23:36

momchil-flex requested a review from tylerflex October 31, 2023 23:37

tylerflex approved these changes Nov 1, 2023

View reviewed changes

tylerflex reviewed Nov 1, 2023

View reviewed changes

momchil-flex force-pushed the momchil/hdf5_string_indent branch from ab81cbe to 6de60b9 Compare November 1, 2023 23:57

tylerflex reviewed Nov 2, 2023

View reviewed changes

momchil-flex force-pushed the momchil/hdf5_string_indent branch from 6de60b9 to 712f140 Compare November 2, 2023 21:40

momchil-flex force-pushed the momchil/hdf5_string_indent branch from 712f140 to e375724 Compare November 3, 2023 17:13

tylerflex reviewed Nov 3, 2023

View reviewed changes

tylerflex approved these changes Nov 3, 2023

View reviewed changes

Setting internal json indent to None, and splitting json string in hdf5

f58e161

momchil-flex force-pushed the momchil/hdf5_string_indent branch from e375724 to f58e161 Compare November 3, 2023 18:06

momchil-flex merged commit c5aec95 into pre/2.5 Nov 3, 2023
14 checks passed

momchil-flex deleted the momchil/hdf5_string_indent branch November 3, 2023 18:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Removing indent in string stored in hdf5 #1228

Removing indent in string stored in hdf5 #1228

momchil-flex commented Oct 31, 2023

tylerflex Nov 1, 2023

momchil-flex commented Nov 2, 2023

tylerflex left a comment

tylerflex Nov 2, 2023

tylerflex Nov 2, 2023

tylerflex Nov 2, 2023

tylerflex Nov 2, 2023

momchil-flex Nov 2, 2023

tylerflex commented Nov 2, 2023

dbochkov-flexcompute commented Nov 2, 2023

tylerflex commented Nov 2, 2023 •

edited

Loading

dbochkov-flexcompute commented Nov 2, 2023

tylerflex commented Nov 2, 2023

momchil-flex commented Nov 2, 2023

tylerflex Nov 3, 2023

tylerflex Nov 3, 2023

momchil-flex Nov 3, 2023

tylerflex Nov 3, 2023

tylerflex left a comment

tylerflex Nov 3, 2023

Removing indent in string stored in hdf5 #1228

Removing indent in string stored in hdf5 #1228

Conversation

momchil-flex commented Oct 31, 2023

Choose a reason for hiding this comment

momchil-flex commented Nov 2, 2023

tylerflex left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tylerflex commented Nov 2, 2023

dbochkov-flexcompute commented Nov 2, 2023

tylerflex commented Nov 2, 2023 • edited Loading

dbochkov-flexcompute commented Nov 2, 2023

tylerflex commented Nov 2, 2023

momchil-flex commented Nov 2, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tylerflex left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tylerflex commented Nov 2, 2023 •

edited

Loading