Skip to content

Add Azure ingestion flow to web console#9437

Merged
vogievetsky merged 4 commits intoapache:masterfrom
implydata:add-azure
Mar 3, 2020
Merged

Add Azure ingestion flow to web console#9437
vogievetsky merged 4 commits intoapache:masterfrom
implydata:add-azure

Conversation

@mcbrewster
Copy link
Contributor

@mcbrewster mcbrewster commented Feb 28, 2020

Screen Shot 2020-02-28 at 1 03 39 PM

Screen Shot 2020-02-28 at 1 03 30 PM

**Please do not merge until support for azure has been merged**

Adds the ability to ingest data from azure via the web console. There is now a tile that allows the user to access the Azure ingestion flow if druid-azure-extensions is in the load list. The default input type will be azure. The user can then put a can then input data using A URI or a Prefix. the resulting IO config changes will be :

  "ioConfig": {
    "type": "index_parallel",
    "inputSource": {
      "type": "azure",
      "prefixes": [
        "azure://containerName"
      ]
    }
  },

or

  "ioConfig": {
    "type": "index_parallel",
    "inputSource": {
      "type": "azure",
      "prefixes": [
        "azure://container-name/path-name.json"
      ]
    }
  },

The user can configure the rest of the spec using the normal web console flow.

Additionally
azure is now available as an input type in the input menu for all flows
Screen Shot 2020-02-28 at 1 05 59 PM

@vogievetsky
Copy link
Contributor

Please do not merge until support for azure has been merged

Can you indicate what this is blocked on?

@mcbrewster
Copy link
Contributor Author

mcbrewster commented Feb 29, 2020 via email

@vogievetsky
Copy link
Contributor

I think you are referring to #9394 and it is merged!


case 'static-azure':
deepSet(spec, 'ioConfig.firehose.type', 'azure');
break;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no need for this upgrade spec clause, there was never a 'static-azure'. This is back compat code but Azure ingestion did not used to exist. You can just remove this change.


case 'azure':
deepSet(spec, 'ioConfig.firehose.type', 'static-azure');
break;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same story as above.

@vogievetsky
Copy link
Contributor

Looks great thank you 👍

@vogievetsky vogievetsky merged commit 99095c4 into apache:master Mar 3, 2020
@vogievetsky vogievetsky deleted the add-azure branch March 3, 2020 19:06
@jihoonson jihoonson added this to the 0.18.0 milestone Mar 26, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants