Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SFN] Enhancements for Map State #10204

Merged
merged 7 commits into from Mar 2, 2024
Merged

Conversation

MEPalma
Copy link
Contributor

@MEPalma MEPalma commented Feb 8, 2024

Motivation

Changes

  • Added distributed legacy iterator components
  • Added related grammar support for processor config in legacy iterator
  • Adjusted the behaviour of ItemsPath
  • Several related changes regarding the misuse of the program's stack
  • Several adjustments to map run logic
  • Added related tests

@MEPalma MEPalma added the semver: patch Non-breaking changes which can be included in patch releases label Feb 8, 2024
@MEPalma MEPalma self-assigned this Feb 8, 2024
@coveralls
Copy link

coveralls commented Feb 8, 2024

Coverage Status

coverage: 83.411%. remained the same
when pulling 274c2b8 on MEP-sfn-dist_legacy_iter_and_others
into 6ec63bf on master.

Copy link

github-actions bot commented Feb 9, 2024

LocalStack Community integration with Pro

    2 files  ± 0      2 suites  ±0   1h 26m 13s ⏱️ - 2m 28s
2 681 tests +10  2 428 ✅ +10  253 💤 ±0  0 ❌ ±0 
2 683 runs  +10  2 428 ✅ +10  255 💤 ±0  0 ❌ ±0 

Results for commit 274c2b8. ± Comparison against base commit 6ec63bf.

♻️ This comment has been updated with latest results.

@MEPalma MEPalma added this to the 3.2 milestone Feb 19, 2024
Copy link
Member

@dominikschubert dominikschubert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, great changes and awesome test coverage!


job.job_program.eval(env)

# TODO: verify behaviour with all of these scenarios.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what would be the strategy to test this? Can this be achieved with certain state machine configuration or would we need to somehow inject errors?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding some state machines that raise error in the MapRun would inform us about this exact behaviour, but this is very connected to the tolerance control aspects of MapRun which will need supporting from both the provider side and the MapRun backend.

@MEPalma MEPalma merged commit 91270a9 into master Mar 2, 2024
29 checks passed
@MEPalma MEPalma deleted the MEP-sfn-dist_legacy_iter_and_others branch March 2, 2024 10:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
semver: patch Non-breaking changes which can be included in patch releases
Projects
None yet
Development

Successfully merging this pull request may close these issues.

bug: stepfunction v2 does not handle distributed processing mode correctly
3 participants