Spring Batch Scaling

Scaling & Parallel

Reference Doc

• Multi-threaded Step
• Parallel Steps
• Remote Chunking of Step
• Partitioning a Step

Multi Threaded step

<step id="loading"> 
  <tasklet task-executor="taskExecutor" throttle-limit="20">...</tasklet>
</step>

• Step executes by reading, processing and writing each chunk of items (each commit interval) in a separate thread of execution. commit interval 단위로 item을 쪼개서 여러 스레드에서 step이 수행됨.
• there is no fixed order for the items to be processed, and a chunk might contain items that are non-consecutive compared to the single-threaded case. 순서 보장되지 않음. 실행되는 chunk 내의 item 역시 sequencial하지 않을 수 있음.
• default throttle limit : 4

Partitioning

<step id="step1.master">
    <partition step="step1" handler="handler"/></step>

<bean class="org.spr...TaskExecutorPartitionHandler">
    <property name="taskExecutor" ref="taskExecutor"/>
    <property name="step" ref="step1" />
    <property name="gridSize" value="10" /></bean>

• The Job is executing on the left hand side as a sequence of Steps, and one of the Steps is labelled as a Master. The Slaves in this picture are all identical instances of a Step, which could in fact take the place of the Master resulting in the same outcome for the Job. 특정한 Step에 대해 M/S 구조로 구동하게 함. Master가 된 step은 slave들에 대해 구동 시그널을 보내고 응답만 받음.
• The Slaves are typically going to be remote services, but could also be local threads of execution. Slave의 경우 remote/local 모두 가능함.
• PartitionHandler : simple RMI remoting, EJB remoting, custom web service, JMS, Java Spaces, shared memory grids (like Terracotta or Coherence), grid execution fabrics (like GridGain) <-- Remote에 대해 지원되는 방식
• gridSize : determines the number of separate step executions to create. default value is 10. 생성되는 slave의 max count.
• spring batch ensure that one execution of slave step per one job. 한 JOB의 step에 대해 slave가 한번만 실행되는 걸 프레임웍 레벨에서 보장함.
• Many articles recommand this solution for solving issues of batch scaling.

Remote partitioning

• Spring Integration을 통한 remote management. (batch integration)
• Using MessageChannelPartitionHandler
• remote worker (remote partitionor)를 구성하고 step에 대한 응답을 받음.
• JMS, AMQP 로 abstraction 제공함. (따로 미들웨어 설치 없이 설정만으로 가능...)