Skip to content

Commit

Permalink
fix: Do not save messages from failed fits
Browse files Browse the repository at this point in the history
Do not save any messages from failed sampling runs.
For example, if sampling starts but cannot continue because attempts
to find appropriate initial values are rejected, there will be some
(logger) messages saved to disk. The presence of these messages
confuses the cache machinery, which assumes the presence of messages
on disk indicates a successful fit. Deleting them avoids the problem.

There remains the problem of retrieving the logger messages associated
with the failed fit. It's possible that this is something someone might
want to do. Future work might address this. Such a user is also free to
use CmdStan.

Closes #452
  • Loading branch information
riddell-stan committed Dec 14, 2020
1 parent 16cfa8a commit 16f36b6
Show file tree
Hide file tree
Showing 2 changed files with 5 additions and 17 deletions.
3 changes: 3 additions & 0 deletions httpstan/views.py
Original file line number Diff line number Diff line change
Expand Up @@ -389,6 +389,9 @@ def _services_call_done(operation: dict, future: asyncio.Future) -> None:
)
logger.critical(message)
operation["result"] = _make_error(message, status=status)
# Delete messages associated with the fit. If initialization
# fails, for example, messages will exist on disk. Remove them.
httpstan.cache.delete_fit(operation["metadata"]["fit"]["name"])
else:
logger.info(f"Operation `{operation['name']}` finished.")
operation["result"] = schemas.Fit().load(operation["metadata"]["fit"])
Expand Down
19 changes: 2 additions & 17 deletions tests/test_sampling_exceptions.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,27 +32,12 @@ async def test_sampling_initialization_failed(api_url: str) -> None:
assert operation["result"].get("code") == 400
assert "Initialization failed." in operation["result"]["message"]

# recover the error messages sent to `logger`
# note that the fit name is retrieved from metadata. If sampling had
# completed without error, it would be available under `result`.
# verify the partial fit has been deleted.
fit_name = operation["metadata"]["fit"]["name"]

# verify operation finished and that fit is available
async with aiohttp.ClientSession() as session:
fit_url = f"{api_url}/{fit_name}"
async with session.get(fit_url) as resp:
assert resp.status == 200

# verify (error) messages are available
fit_bytes_ = await helpers.fit_bytes(api_url, fit_name)
assert isinstance(fit_bytes_, bytes)
messages = helpers.decode_messages(fit_bytes_)

assert len(messages) > 100

# first message should be an "Rejecting initial value" message.
error_message = messages[0]["values"].pop()
assert "Rejecting initial value" in error_message
assert resp.status == 404


@pytest.mark.asyncio
Expand Down

0 comments on commit 16f36b6

Please sign in to comment.