Skip to content

Simulation _set_data method improperly removes file link #154

@anth-volk

Description

@anth-volk

The code causing the issue begins at

if isinstance(self.options.data, str):

This prevents all simulations from running properly, thereby preventing launch of API v2.

Full trace from API:

INFO:     169.254.169.126:22308 - "POST /simulate/economy/comparison HTTP/1.1" 500 Internal Server Error
Initialising

ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/app/projects/app/.venv/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/app/projects/app/.venv/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
  File "/app/projects/app/.venv/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/app/projects/app/.venv/lib/python3.11/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/app/projects/app/.venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/app/projects/app/.venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/app/projects/app/.venv/lib/python3.11/site-packages/starlette/middleware/base.py", line 176, in __call__
    with recv_stream, send_stream, collapse_excgroups():
  File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/app/projects/app/.venv/lib/python3.11/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
    raise exc
  File "/app/projects/app/.venv/lib/python3.11/site-packages/starlette/middleware/base.py", line 178, in __call__
    response = await self.dispatch_func(request, call_next)
  File "/app/projects/app/.venv/lib/python3.11/site-packages/policyengine_api/fastapi/opentelemetry/middleware.py", line 47, in __call__
    return await call_next(request)
  File "/app/projects/app/.venv/lib/python3.11/site-packages/starlette/middleware/base.py", line 156, in call_next
    raise app_exc
  File "/app/projects/app/.venv/lib/python3.11/site-packages/starlette/middleware/base.py", line 141, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "/app/projects/app/.venv/lib/python3.11/site-packages/opentelemetry/instrumentation/asgi/__init__.py", line 743, in __call__
    await self.app(scope, otel_receive, otel_send)
  File "/app/projects/app/.venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/app/projects/app/.venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/app/projects/app/.venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/app/projects/app/.venv/lib/python3.11/site-packages/starlette/routing.py", line 73, in app
    response = await f(request)
  File "/app/projects/app/.venv/lib/python3.11/site-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
  File "/app/projects/app/.venv/lib/python3.11/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
  File "/app/libs/policyengine-simulation-api/src/policyengine_api/simulation_api/simulation.py", line 21, in simulate
    simulation = Simulation(**model.model_dump())
  File "/app/projects/app/.venv/lib/python3.11/site-packages/policyengine/simulation.py", line 92, in __init__
    self._set_data()
  File "/app/projects/app/.venv/lib/python3.11/site-packages/policyengine/simulation.py", line 158, in _set_data
    self.options.data = Dataset.from_file(
  File "/app/projects/app/.venv/lib/python3.11/site-packages/policyengine_core/data/dataset.py", line 412, in from_file
    with h5py.File(file_path, "r") as f:
  File "/app/projects/app/.venv/lib/python3.11/site-packages/h5py/_hl/files.py", line 564, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
  File "/app/projects/app/.venv/lib/python3.11/site-packages/h5py/_hl/files.py", line 238, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 102, in h5py.h5f.open
FileNotFoundError: [Errno 2] Unable to synchronously open file (unable to open file: name = 'gs:/policyengine-us-data/cps_2023.h5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)

INFO:     169.254.169.126:22310 - "POST /ping HTTP/1.1" 200 OK
      INFO   169.254.169.126:23504 - "GET /ping/alive HTTP/1.1" 200
      INFO   169.254.169.126:54783 - "GET /ping/alive HTTP/1.1" 200
      INFO   169.254.169.126:22766 - "POST /ping HTTP/1.1" 200

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions