[Proposal] Move Task to move segments while migrating deep storages

[niketh]

Introduce a new task - Move Task which can move segments and insert metadata correctly while moving deep storages.

@gianm

{
    "type": "move",
    "id": <task_id>,
    "dataSource": <task_datasource>,
    "destination": {
        "overlord": <url>,
        "ioConfig" : {
              "type" : "hadoop",
               "outputSpec" : {
                   "paths" : "hdfs://xxx/yyy/zzz"
               }
          }
       }
    "interval": <interval to specify segments to be merged>,
    "tuningConfig" <move task tuningConfig>,
    "context": <task context>
}

[jihoonson]

We already have the moveTask for this purpose. What's the difference?

[niketh]

@jihoonson I didn't find it in the documentation http://druid.io/docs/latest/ingestion/tasks.html.
I saw the code some more, some problems with existing implementation:

1. It directly runs as a Middle Manager local task, no map reduce option.
2. Doesn't support segments on HDFS.

[jihoonson]

Yeah, we should document and improve it.

[niketh]

Agreed. I will take a stab at it.

[stale]

This issue has been marked as stale due to 280 days of inactivity. It will be closed in 2 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@druid.apache.org list. Thank you for your contributions.

[stale]

This issue has been marked as stale due to 280 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@druid.apache.org list. Thank you for your contributions.

[stale]

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.