Remove cloud code #808
Closed
Remove cloud code #808
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…nager is called only once where the hardware description information is sent.
Conflicts: stratosphere-runtime/src/main/java/eu/stratosphere/nephele/instance/cluster/ClusterManager.java stratosphere-runtime/src/main/java/eu/stratosphere/nephele/taskmanager/TaskManager.java stratosphere-runtime/src/test/java/eu/stratosphere/nephele/instance/cluster/ClusterManagerTest.java
…stanceTypeDescriptionTypeFactory, PendingRequestsMap
…ultInstanceManager.
…ied DefaultScheduler.
Conflicts: stratosphere-clients/src/main/java/eu/stratosphere/client/minicluster/NepheleMiniCluster.java stratosphere-compiler/src/main/java/eu/stratosphere/compiler/PactCompiler.java stratosphere-compiler/src/main/java/eu/stratosphere/compiler/dag/DataSourceNode.java stratosphere-compiler/src/main/java/eu/stratosphere/compiler/dataproperties/RequestedGlobalProperties.java stratosphere-compiler/src/main/java/eu/stratosphere/compiler/operators/GroupReduceWithCombineProperties.java stratosphere-compiler/src/main/java/eu/stratosphere/compiler/operators/PartialGroupProperties.java stratosphere-compiler/src/main/java/eu/stratosphere/compiler/plan/PlanNode.java stratosphere-core/src/main/java/eu/stratosphere/configuration/ConfigConstants.java stratosphere-runtime/src/main/java/eu/stratosphere/nephele/executiongraph/ExecutionGraph.java stratosphere-runtime/src/main/java/eu/stratosphere/nephele/executiongraph/ExecutionGroupVertex.java stratosphere-runtime/src/main/java/eu/stratosphere/nephele/instance/local/LocalInstanceManager.java stratosphere-runtime/src/main/java/eu/stratosphere/nephele/jobmanager/scheduler/DefaultScheduler.java stratosphere-runtime/src/main/java/eu/stratosphere/nephele/jobmanager/scheduler/local/LocalScheduler.java stratosphere-runtime/src/main/java/eu/stratosphere/nephele/managementgraph/ManagementVertex.java stratosphere-runtime/src/main/java/eu/stratosphere/pact/runtime/sort/AsynchronousPartialSorter.java stratosphere-runtime/src/main/java/eu/stratosphere/pact/runtime/sort/AsynchronousPartialSorterCollector.java stratosphere-runtime/src/main/java/eu/stratosphere/pact/runtime/sort/CombiningUnilateralSortMerger.java stratosphere-runtime/src/main/java/eu/stratosphere/pact/runtime/sort/UnilateralSortMerger.java stratosphere-runtime/src/main/java/eu/stratosphere/pact/runtime/task/DataSinkTask.java stratosphere-runtime/src/main/java/eu/stratosphere/pact/runtime/task/GroupReduceCombineDriver.java stratosphere-runtime/src/main/java/eu/stratosphere/pact/runtime/task/MatchDriver.java stratosphere-runtime/src/main/java/eu/stratosphere/pact/runtime/task/chaining/ChainedCombineDriver.java stratosphere-runtime/src/test/java/eu/stratosphere/nephele/instance/cluster/HostInClusterTest.java stratosphere-runtime/src/test/java/eu/stratosphere/pact/runtime/sort/AsynchonousPartialSorterITCase.java stratosphere-runtime/src/test/java/eu/stratosphere/pact/runtime/sort/CombiningUnilateralSortMergerITCase.java stratosphere-runtime/src/test/java/eu/stratosphere/pact/runtime/sort/UnilateralSortMergerITCase.java stratosphere-runtime/src/test/java/eu/stratosphere/pact/runtime/task/CombineTaskExternalITCase.java stratosphere-runtime/src/test/java/eu/stratosphere/pact/runtime/task/CombineTaskTest.java stratosphere-runtime/src/test/java/eu/stratosphere/pact/runtime/task/DataSinkTaskTest.java stratosphere-runtime/src/test/java/eu/stratosphere/pact/runtime/task/ReduceTaskExternalITCase.java stratosphere-runtime/src/test/java/eu/stratosphere/pact/runtime/task/ReduceTaskTest.java stratosphere-runtime/src/test/java/eu/stratosphere/pact/runtime/test/util/DriverTestBase.java stratosphere-test-utils/src/main/java/eu/stratosphere/test/util/AbstractTestBase.java stratosphere-tests/src/test/java/eu/stratosphere/test/broadcastvars/KMeansIterativeNepheleITCase.java stratosphere-tests/src/test/java/eu/stratosphere/test/exampleRecordPrograms/ComputeEdgeDegreesITCase.java stratosphere-tests/src/test/java/eu/stratosphere/test/exampleRecordPrograms/EnumTrianglesOnEdgesWithDegreesITCase.java stratosphere-tests/src/test/java/eu/stratosphere/test/exampleRecordPrograms/GlobalSortingITCase.java stratosphere-tests/src/test/java/eu/stratosphere/test/exampleRecordPrograms/GlobalSortingMixedOrderITCase.java stratosphere-tests/src/test/java/eu/stratosphere/test/exampleRecordPrograms/GroupOrderReduceITCase.java stratosphere-tests/src/test/java/eu/stratosphere/test/exampleRecordPrograms/KMeansStepITCase.java stratosphere-tests/src/test/java/eu/stratosphere/test/exampleRecordPrograms/TPCHQuery3ITCase.java stratosphere-tests/src/test/java/eu/stratosphere/test/exampleRecordPrograms/TPCHQuery3WithUnionITCase.java stratosphere-tests/src/test/java/eu/stratosphere/test/exampleRecordPrograms/TPCHQuery4ITCase.java stratosphere-tests/src/test/java/eu/stratosphere/test/exampleRecordPrograms/TPCHQuery9ITCase.java stratosphere-tests/src/test/java/eu/stratosphere/test/exampleRecordPrograms/TPCHQueryAsterixITCase.java stratosphere-tests/src/test/java/eu/stratosphere/test/exampleRecordPrograms/TeraSortITCase.java stratosphere-tests/src/test/java/eu/stratosphere/test/exampleRecordPrograms/WebLogAnalysisITCase.java stratosphere-tests/src/test/java/eu/stratosphere/test/exampleRecordPrograms/WordCountITCase.java stratosphere-tests/src/test/java/eu/stratosphere/test/exampleRecordPrograms/WordCountUnionReduceITCase.java stratosphere-tests/src/test/java/eu/stratosphere/test/iterative/CoGroupConnectedComponentsITCase.java stratosphere-tests/src/test/java/eu/stratosphere/test/iterative/ConnectedComponentsITCase.java stratosphere-tests/src/test/java/eu/stratosphere/test/iterative/ConnectedComponentsWithDeferredUpdateITCase.java stratosphere-tests/src/test/java/eu/stratosphere/test/iterative/ConnectedComponentsWithSolutionSetFirstITCase.java stratosphere-tests/src/test/java/eu/stratosphere/test/iterative/DanglingPageRankITCase.java stratosphere-tests/src/test/java/eu/stratosphere/test/iterative/DeltaPageRankITCase.java stratosphere-tests/src/test/java/eu/stratosphere/test/iterative/IterationTerminationWithTerminationTail.java stratosphere-tests/src/test/java/eu/stratosphere/test/iterative/IterationTerminationWithTwoTails.java stratosphere-tests/src/test/java/eu/stratosphere/test/iterative/IterationWithAllReducerITCase.java stratosphere-tests/src/test/java/eu/stratosphere/test/iterative/IterationWithChainingITCase.java stratosphere-tests/src/test/java/eu/stratosphere/test/iterative/IterationWithUnionITCase.java stratosphere-tests/src/test/java/eu/stratosphere/test/iterative/IterativeKMeansITCase.java stratosphere-tests/src/test/java/eu/stratosphere/test/iterative/KMeansTutorialExampleITCase.java stratosphere-tests/src/test/java/eu/stratosphere/test/iterative/LineRankITCase.java stratosphere-tests/src/test/java/eu/stratosphere/test/iterative/PageRankITCase.java stratosphere-tests/src/test/java/eu/stratosphere/test/iterative/nephele/ConnectedComponentsNepheleITCase.java stratosphere-tests/src/test/java/eu/stratosphere/test/iterative/nephele/DanglingPageRankNepheleITCase.java stratosphere-tests/src/test/java/eu/stratosphere/test/iterative/nephele/DanglingPageRankWithCombinerNepheleITCase.java stratosphere-tests/src/test/java/eu/stratosphere/test/iterative/nephele/IterationWithChainingNepheleITCase.java stratosphere-tests/src/test/java/eu/stratosphere/test/operators/UnionSinkITCase.java
|
Nice. That one is also a contender for "biggest pull request" ;-) I think we will merge that after the 0.5 release, as it changes quite a bit of the model |
|
+1 I think we should go over this PR as soon as 0.5 is out. Looking forward to it. :-) |
|
Yes, it will be one of the first after the 0.5 release... |
|
I am trying to merge this in the 0.6-SNAPSHOT master |
|
+1 Sent from my iPhone On 31.05.2014, at 19:25, Stephan Ewen notifications@github.com wrote: I am trying to merge this in the 0.6-SNAPSHOT master — |
|
This changes a lot:
|
|
That one will keep me busy tonight ;-) |
|
The most of it should only be of cosmetic nature. Next time, I'll split the
|
|
The pull request removed the "connection IDs" from the edges. I think they are important to decide what can be multiplexed though the same TCP connection and what cannot. |
|
I am re-adding them |
|
@StephanEwen Yes, the netstack relies on the connection IDs. |
|
This has been merged in the new repo in commits 86d206c...b4b633e. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Removed legacy Nephele cloud code. Among others, all InstanceType related code is removed. Furthermore, a simple slot system for the TaskManager is introduced. Each TaskManager has a set of slots. Either the number is set in the configuration or the number of cores is taken as the default value. This implies that the intra-node parallelism was abandoned. Additionally a relative memory assignment was implemented. During the compiling process, for each task its memory share is calculated. Once deployed this value is translated with respect to the available memory on the TaskManager.