Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug: very slow export with model_attributes_to_shapefile #1927

Closed
RyanConway91 opened this issue Aug 23, 2023 · 2 comments
Closed

bug: very slow export with model_attributes_to_shapefile #1927

RyanConway91 opened this issue Aug 23, 2023 · 2 comments
Assignees
Labels
Milestone

Comments

@RyanConway91
Copy link

Describe the bug
Not a true bug, just a performance FYI
'model_attributes_to_shapefile' gets hung up and did not write anything after ~20min for my large disv model. If I write my own version, its works pretty fast (<1 min)

f_out = r'C:\offline\mcwhpp\junk\mdl_shps\whpp.shp'
sim_ws = r'V:\23\62\1462\multi_comm_WHPP\mf6\calibration\current'
sim = fp.mf6.MFSimulation.load(sim_ws= sim_ws)
ml = sim.get_model('local')
grb = fp.mf6.utils.MfGrdFile(r'%s\mcwhpp_m.dis.grb'%sim_ws)
modelgrid = grb.modelgrid
epsg = 26915
modelgrid.set_coord_info(crs = pyproj.CRS.from_epsg(epsg))
if False:
    #never does anything
    fp.export.shapefile_utils.model_attributes_to_shapefile(path = f_out,ml = ml, modelgrid =modelgrid,package_names = ['npf'])
if True:
    #my version... works in ~1min
    dir_out = r'C:\offline\mcwhpp\junk\mdl_shps'
    d_pks = {'npf':{'k':ml.npf.k.array,'k33':ml.npf.k33.array}}
    for pk, d_att in d_pks.items():
        array_dict = {}
        for att,a in d_att.items():
            a_ls = a.reshape((modelgrid.nlay,modelgrid.ncpl))
            for l,_a in enumerate(a_ls):
                array_dict['%s_L%i'%(att,l+1)] = _a
        fp.export.shapefile_utils.write_grid_shapefile(r'%s\%s_grd.shp'%(dir_out,pk),mg = modelgrid, array_dict = array_dict)

Expected behavior
I expect 'model_attributes_to_shapefile' to work more quickly

Desktop (please complete the following information):

  • OS: Windows 10
  • Version '3.5.0.dev0'

Additional context
Add any other context about the problem here.

@spaulins-usgs spaulins-usgs self-assigned this Aug 29, 2023
@spaulins-usgs
Copy link
Contributor

@RyanConway91, so far I have not been able to reproduce this. I used "test050_create_tests_circle_island" from our tests in autotest\regression\test_mf6.py, and increased the number of layers to 400, so that the model has over 2,000,000 cells.

Exporting with "model_attributes_to_shapefile" took 35 seconds. Exporting with the code you provided above took 17 seconds. Note that "model_attributes_to_shapefile" exports more data than your code does, exported shapefiles are 119 MB vs 82 MB ("model_attributes_to_shapefile" also exports icelltype). In this case "model_attributes_to_shapefile" is running a little slower than your code (3.4 MB/sec vs 4.8 MB/sec), though nowhere near what you are describing.

Can you provide an example project to help me reproduce the problem?

@wpbonelli wpbonelli added this to the 3.7.0 milestone May 1, 2024
@wpbonelli
Copy link
Contributor

Feel free to reopen if needed @RyanConway91

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants