Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automated testing of TimeSteward callers #5

Open
elidupree opened this issue Aug 21, 2016 · 3 comments
Open

Automated testing of TimeSteward callers #5

elidupree opened this issue Aug 21, 2016 · 3 comments

Comments

@elidupree
Copy link
Owner

elidupree commented Aug 21, 2016

There are various ways that TimeSteward callers can misbehave, which we should find ways to audit for.

  • Using nondeterministic code in predictors or events.
  • Using field types that have lossy serialization or don't have an exact match between Eq and serialization equality.
  • Using unsafe_now() improperly.
  • Using (column, event, predictor) types that are not included in Basics::IncludedTypes.
  • Using nonrandom data in (column, event, predictor) ids.
@elidupree
Copy link
Owner Author

One of the important techniques for testing determinism is to run 2 instances of the same simulation on different devices, and continuously verify that they remain synchronized – even with network delay affecting the fiat events issued by each device. This requires a full TimeSteward implementation, and also some additional features we haven't settled on yet.

@elidupree
Copy link
Owner Author

simply_synchronized is now working and successfully caught me using HashSet inappropriately. There is much more work to be done in this area – including making the panic messages more detailed and understandable – but I consider this sufficient for MVP.

@elidupree
Copy link
Owner Author

With the API changes, the old testing is no longer applicable, and there are many new tests that need to be done.

My idea is to make an auditing TimeSteward – a separate TimeSteward implementor that does costly operations to detect as many possible incorrect behaviors as possible. I'm thinking it would also be able to take a replay – a record of fiat events plus a record of what order all events were run – and rerun the simulation in that order, even if the original simulation was run by a real-time TimeSteward instead of the auditing one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant