Skip to content

Stream State initial implementation #119

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 39 commits into from
May 25, 2023
Merged

Stream State initial implementation #119

merged 39 commits into from
May 25, 2023

Conversation

@peter-quix
Copy link
Contributor

Implemented the stream states in C# and Python.

  • C# support generic and non-generic variant, while python is only supporting generic variant for now.
  • State is saved OnCommitting.
  • State is revoked when stream is revoked.

Known issues:

  • Potential improvement could be done by adding staging support in IStateStorage. Stage with OnCommitting, Finalize with OnCommitted, to reduce chance of unsuccessful storage save to minimum offered by storage limitations.
  • When a consumed stream is closed, revocation doesn't include it in the list. this can cause saving state even when shouldn't
  • Possibly threading issues when modifying a state from multiple threads. Attempted to reproduce using unit tests in C# and python without luck. Left without locking implemented for now

Peter Nagy added 26 commits May 8, 2023 16:23
State improvements
c6d0978 
- Move tests from MS tests to xunit
- Introduce State, which is a key-val in memory state backed by another persisted state
- Minor fix for StateValue to properly compare
- FileStorage is no longer case sensitive
- Add Case sensitivity to IStateStorage
- Add Count to IStateStorage
Implement TopicState
410e0ad 
- Implement TopicState
- Add Topic information to TelemetryKafkaConsumer
- Add tests
In c# integration tests use platform specific image
487a862
Refactor State
e0e7e26 
- Added State<T> 
- Refactored TopicState to use State<T>
- Added TopicState
- Added PrefixedLoggerFactory
More StreamState work
ef27433 
- Moved StreamState next to StreamConsumer
- Removed unused properties all around
- Implement double-lock pattern where relevant
- Hook up StreamState, so it is actually usable
Up version to 0.5.3 and dev 1
54a1021
add management API on top of stream states and also add unit test
ffd4f14
Update InteropGenerator
ba24ff6 
- Change to Logger from Console.WriteLine
- Handle unsupported types better, filter them out sooner
Rework state managers
5db5498 
- Add App state manager to allow overriding the state storage
- Add missing management functions such as get/delete 
- Some WIP for python
Remove TopicState
b022ae6
Change how prefixing works for sub streams
96e4230
Simplify the naming for localfilestorage
33b9422 
Will deal with topic/app states using another method, but for now we don't need any prefixing
Fix documentation for raw_topic
fa6879e
InMemoryStorage exposed in python
263389d 
- some reference fixes to be relative
- AppStateManager exposed
up to 0.5.4.dev1
5580997
WIP
9ced32a
Fix package imports at places to avoid using 2 version of the lib
e49a484
Several fixed to get state working in python
ad19cb3
dev2
5593c86
Fix reference type values
3bf6c31 
- Add support for reference type values
- Add support for handling invalid type retrieval
- Handle interop keyerror
Dev3
1ac86ac
Dev 4 after rebase
c3021b0
Fix same type not being accepted for state
d7aa5e0
Add some unit test around possible threading problems
27c09d7
Add option to reset state
3ceefc3
update to dev 5
819a167
@peter-quix peter-quix marked this pull request as draft May 9, 2023 22:52
harisbotic
harisbotic previously approved these changes May 17, 2023
View reviewed changes
Copy link
Contributor

@harisbotic harisbotic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good expect from few renames or cleanups that I would do

@peter-quix peter-quix marked this pull request as ready for review May 22, 2023 15:16
@harisbotic harisbotic merged commit 4776fe0 into main May 25, 2023
@harisbotic harisbotic deleted the feature/state-helpers branch May 25, 2023 09:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@harisbotic harisbotic harisbotic approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@peter-quix @harisbotic