Skip to content

[BEAM-3900] Improvements in Metrics translation, composite transforms, tests and win. API#5783

Closed
VaclavPlajt wants to merge 3718 commits intoapache:dsl-euphoriafrom
seznam:rebased-dsl-euphoria
Closed

[BEAM-3900] Improvements in Metrics translation, composite transforms, tests and win. API#5783
VaclavPlajt wants to merge 3718 commits intoapache:dsl-euphoriafrom
seznam:rebased-dsl-euphoria

Conversation

@VaclavPlajt
Copy link
Contributor

@VaclavPlajt VaclavPlajt commented Jun 27, 2018

List of improvements:

  • Larger portfolio of Euphoria test suite was enabled.
  • Translation of Euphoria Accumulators to Beam Metrics was added.
  • Euphoria windowing API adapted to Beam windowing.
  • Operator's API now consumes type hints to allow set coders automatically.
  • Composite transforms can be defined through Euphoria API.
  • Cleanup: Flipping tests fixed, Euphoria executor concept removed, two project modules where merged
    R: @jbonofre

Follow this checklist to help us incorporate your contribution quickly and easily:

  • Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

It will help us expedite review of your Pull Request if you tag someone (e.g. @username) to look at it.

Post-Commit Tests Status (on master branch)

Lang SDK Apex Dataflow Flink Gearpump Spark
Go Build Status --- --- --- --- ---
Java Build Status Build Status Build Status Build Status Build Status Build Status
Python Build Status --- Build Status
Build Status
--- --- ---

@kennknowles
Copy link
Member

We have turned on autoformatting of the codebase, which causes small conflicts across the board. You can probably safely rebase and just keep your changes. Like this:

$ git rebase
... see some conflicts
$ git diff
... confirmed that the conflicts are just autoformatting
... so we can just keep our changes are do our own autoformat
$ git checkout --theirs --
$ git add -u
$ git rebase --continue
$ ./gradlew spotlessJavaApply

Please ping me if you run into any difficulty.

@mareksimunek mareksimunek force-pushed the rebased-dsl-euphoria branch from 1ab7782 to 131d558 Compare July 9, 2018 07:57
@VaclavPlajt VaclavPlajt changed the title [BEAM-3900] Rebased dsl euphoria [BEAM-3900] Improvements in Metrics translation, composite transforms, tests and windowing AP Jul 10, 2018
@VaclavPlajt VaclavPlajt changed the title [BEAM-3900] Improvements in Metrics translation, composite transforms, tests and windowing AP [BEAM-3900] Improvements in Metrics translation, composite transforms, tests and windowing API Jul 11, 2018
@VaclavPlajt VaclavPlajt changed the title [BEAM-3900] Improvements in Metrics translation, composite transforms, tests and windowing API [BEAM-3900] Improvements in Metrics translation, composite transforms, tests and win. API Jul 11, 2018
@VaclavPlajt
Copy link
Contributor Author

please retest this

@mareksimunek
Copy link
Contributor

Run Java PreCommit

je-ik and others added 22 commits August 17, 2018 11:58
…er partitioning removal, still some tests failing
VaclavPlajt and others added 24 commits August 17, 2018 12:01
…st, FilterTest enabled.

Small modifications where necessary to make them work.
TopPerKeyTest could not be enabled since TopPerKey is translated into unsupported ReduceStateByKey operator.
WindowingTest need to be redesigned. WatermarkTest was removed.
…executor after rebase above 'dsl-euphoria'.
…utorContext` renamed to `TranslationContext`.
…sess from `euphoria-beam` moved to `...euphoria.core.translate` package.
All `build.gradle` files in euphoria modules adapted. Some compilation warnings removed, `@SuppressFBWarnings` removed due to licensing issues, unused utils class `OperatorTranslator` removed.
…Euphoria-core build adapted after rebase on beam/master.
@dmvk dmvk force-pushed the rebased-dsl-euphoria branch from 68fa362 to 69c912e Compare August 17, 2018 11:40
@dmvk dmvk force-pushed the rebased-dsl-euphoria branch from 69c912e to a34d5d3 Compare August 17, 2018 11:48
@akedin
Copy link
Contributor

akedin commented Oct 9, 2018

Is this PR in progress or should we close it as stale?

@VaclavPlajt
Copy link
Contributor Author

Hi,
this PR definitely stall. Thanks for your comment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants