Azure Log Analytics Workspace Task Logs Ingestion #10273

kumargauravin · 2020-08-10T09:20:33Z

Initial version of code for review.
Needs to be tested with real account which has permissions which I do not have at this time.
Some ToDo left added as comments.

boring-cyborg · 2020-08-10T09:20:37Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
Here are some useful points:

Pay attention to the quality of your code (flake8, pylint and type annotations). Our pre-commits will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
Consider using Breeze environment for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://apache-airflow-slack.herokuapp.com/

ephraimbuddy · 2020-08-10T09:47:09Z

airflow/providers/microsoft/azure/hooks/laws.py

Can you check if we can do this with any of these libraries:
https://pypi.org/project/azure-mgmt-loganalytics/0.7.0/
https://pypi.org/project/azure-loganalytics/0.1.0/

I see a lot of breaking changes commits per version of these libraries.
Further, the real thing to look for a library would be if it handles the split functionality due to API limit. Which I did not found here. But will check more. For our purpose and full control though, I think this will provide more control. Let me know your thoughts.

@mik-laj @turbaszek what do you think?

@kumargauravin , I actually don't think it's ok to provide access key in the way you're currently doing it. We have connection table to manage authentication information. You can take inspiration from the other hooks in the Azure package. That's what I think. :\

OK. That is a nice idea. Can you refer to anyone and I will try to work in same way. Also for pylint, I may need some help setting up on that breeze setup.

Check the video here https://github.com/apache/airflow/blob/master/BREEZE.rst for setting up breeze environment.

Try using the Azure Batch hook https://github.com/apache/airflow/blob/master/airflow/providers/microsoft/azure/hooks/azure_batch.py

In the case of Stackdriver, we used configurations via airflow.cfg. It is easier to use, but it is not a generally accepted rule.

ephraimbuddy · 2020-08-10T09:51:54Z

You need to install pre-commit in your development environment https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#pre-commit-hooks

kumargauravin · 2020-08-10T23:51:57Z

Hi, Let’s set priorities. 1. Pylint: I will go with the video and see what I understand. 2. Complete ToDo sections and how we can test it. 3. What else needs to be done in order to use the PI. 4. Anything else we require to add to PR. Any other SOP issues you see! 5. Movement of config to table as per example provided. Can you contribute also like in any one points like pylint will be helpful. Add any missing things and let’s get this close and merged so that we can use!!! Regards, Kumar

On Mon, Aug 10, 2020 at 4:18 PM Kamil Breguła ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In airflow/providers/microsoft/azure/hooks/laws.py <#10273 (comment)>: > + self.conn_id = remote_conn_id + self.account_id = account_id + self.access_key = access_key + self.table_name = table_name + + def build_signature(self, date, content_length, method, content_type, resource): + x_headers = 'x-ms-date:' + date + string_to_hash = method + "\n" + str(content_length) + "\n" + content_type + "\n" + x_headers + "\n" + resource + bytes_to_hash = bytes(string_to_hash, encoding="utf-8") + decoded_key = base64.b64decode(self.access_key) + encoded_hash = base64.b64encode( + hmac.new(decoded_key, bytes_to_hash, digestmod=hashlib.sha256).digest() + ).decode() + authorization = "SharedKey {}:{}".format(self.account_id, encoded_hash) + return authorization + In the case of Stackdriver, we used configurations via airflow.cfg. It is easier to use, but it is not a generally accepted rule. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10273 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADK64R7FNRVPNAU72DSAPTLSABP2VANCNFSM4PZYJCCQ> .

-- Regards, Kumar Gaurav +1 469 354 2101

kumargauravin · 2020-08-23T04:16:12Z

@feluelle @ephraimbuddy Request to check once and suggest. I want to complete once cycle of build tests and then continue to work.

feluelle · 2020-08-26T09:45:09Z

airflow/providers/microsoft/azure/hooks/laws.py

+#
+"""
+This module contains integration with Azure LAWS.
+


Suggested change

feluelle · 2020-08-26T09:50:30Z

airflow/providers/microsoft/azure/hooks/laws.py

+        uri = (
+            'https://' + self.account_id + '.ods.opinsights.azure.com' + resource + '?api-version=2016-04-01'
+        )


Suggested change

uri = (

'https://' + self.account_id + '.ods.opinsights.azure.com' + resource + '?api-version=2016-04-01'

)

uri = f'https://{self.account_id}.ods.opinsights.azure.com{resource}?api-version=2016-04-01'

feluelle · 2020-08-26T09:56:50Z

airflow/providers/microsoft/azure/hooks/laws.py

+            response = requests.post(uri, data=body, headers=headers, verify=ssl_verify)
+            response.raise_for_status()


I think thats what a basic HttpHook also does? WDYT of inheriting from HttpHook instead of BaseHook? There are a lot of similiarities.

OK will try that out. Thanks for your comments.

Did that work? Or haven't you tried that yet?

stale · 2020-10-12T00:01:51Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

feluelle · 2020-10-22T08:46:48Z

Hey @kumargauravin how is this going? Do you need any help?

The tests are failing because you must add tests for the provider class you added. The config test is also failing. Can you fix that?

kumargauravin · 2020-10-22T13:41:13Z

hi, Yes not much knowledge about testing if someone can help getting this pass I will get idea. Regards, Kumar

…

On Thu, Oct 22, 2020 at 2:17 PM Felix Uellendall ***@***.***> wrote: Hey @kumargauravin <https://github.com/kumargauravin> how is this going? Do you need any help? The tests are failing because you must add tests for the provider class you added. The config test is also failing. Can you fix that? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10273 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADK64R4L2S7CFS6XGUMCPT3SL7WQRANCNFSM4PZYJCCQ> .

-- Regards, Kumar Gaurav +1 469 354 2101

feluelle · 2020-10-28T08:43:58Z

For testing the task handler you can check out test_es_task_handler.py for example. It contains unittest by simply mocking elasticsearch. You would have to mock azlaws. You can do this by simply using unittest.mock.

kumargauravin · 2020-10-29T13:09:00Z

Sure Felix, Haven't tried will try soon and update. Regards, Kumar

…

On Wed, Oct 28, 2020 at 2:14 PM Felix Uellendall ***@***.***> wrote: For testing the task handler you can check out test_es_task_handler.py <https://github.com/apache/airflow/blob/master/tests/providers/elasticsearch/log/test_es_task_handler.py> for example. It contains unittest by simply mocking elasticsearch. You would habe to mock azlaws. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10273 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADK64R73WNSLQPAHZMTBWMDSM7KV3ANCNFSM4PZYJCCQ> .

-- Regards, Kumar Gaurav +1 469 354 2101

stale · 2020-12-25T12:05:42Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

malthe · 2021-12-16T11:06:15Z

@kumargauravin I am interested in this functionality. Does the code basically work and what's needed from here is to get the pull request into good merge shape (i.e., sort out whatever issues have been mentioned).

kumargauravin · 2021-12-16T19:02:52Z

Test cases if you could please I am occupied with something else. Thanks in advance.

On Thu, Dec 16, 2021 at 6:06 AM Malthe Borch ***@***.***> wrote: @kumargauravin <https://github.com/kumargauravin> I am interested in this functionality. Does the code basically work and what's needed from here is to get the pull request into good merge shape (i.e., sort out whatever issues have been mentioned). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10273 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADK64R34MYROAQFNLH35EG3URHB3HANCNFSM4PZYJCCQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Regards, Kumar Gaurav +1 469 354 2101

kumargauravin · 2021-12-16T19:03:17Z

Rest some suggestion were thr it may have to be rebased. On Thu, Dec 16, 2021 at 2:02 PM Kumar Gaurav ***@***.***> wrote:

Test cases if you could please I am occupied with something else. Thanks in advance. On Thu, Dec 16, 2021 at 6:06 AM Malthe Borch ***@***.***> wrote: > @kumargauravin <https://github.com/kumargauravin> I am interested in > this functionality. Does the code basically work and what's needed from > here is to get the pull request into good merge shape (i.e., sort out > whatever issues have been mentioned). > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#10273 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ADK64R34MYROAQFNLH35EG3URHB3HANCNFSM4PZYJCCQ> > . > Triage notifications on the go with GitHub Mobile for iOS > <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> > or Android > <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. > > -- Regards, Kumar Gaurav +1 469 354 2101

-- Regards, Kumar Gaurav +1 469 354 2101

boring-cyborg bot added area:logging provider:microsoft-azure Azure-related issues labels Aug 10, 2020

ephraimbuddy reviewed Aug 10, 2020

View reviewed changes

turbaszek requested a review from feluelle August 12, 2020 06:38

feluelle changed the title ~~Azure Log Aanalytics Workspace Task Logs Ingestion~~ Azure Log Analytics Workspace Task Logs Ingestion Aug 13, 2020

base added 6 commits August 26, 2020 01:24

Initial Commit [Not Working] [File Structure Changes]

c3b0d10

File Changes for LAWS - 001

ecce738

File Changes for LAWS - 002 - Fix error

7d1d4b0

Almost Working. Need LAWS to test. [ Good to Review ] [ ToDo Pending]

adfbe60

Missed this file before.

4759535

[pylint] Changes to fix pylint

0de6fe2

kumargauravin force-pushed the master branch from b2d86c8 to 0de6fe2 Compare August 26, 2020 06:26

[airflow] kumar gaurav: changes per static-changes

f87fd6a

feluelle reviewed Aug 26, 2020

View reviewed changes

Kumar Gaurav added 2 commits August 26, 2020 19:14

[airflow] kumar gaurav: correction in operators-and-hooks-ref.rst

2ba173c

[airflow] kumar gaurav: add azlaws to spelling

cb310bd

stale bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Oct 12, 2020

stale bot removed the stale Stale PRs per the .github/workflows/stale.yml policy file label Oct 22, 2020

stale bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Dec 25, 2020

github-actions bot closed this Mar 15, 2021

		response = requests.post(uri, data=body, headers=headers, verify=ssl_verify)
		response.raise_for_status()

Azure Log Analytics Workspace Task Logs Ingestion #10273

Azure Log Analytics Workspace Task Logs Ingestion #10273

Uh oh!

Conversation

kumargauravin commented Aug 10, 2020

Uh oh!

boring-cyborg bot commented Aug 10, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ephraimbuddy Aug 10, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ephraimbuddy commented Aug 10, 2020

Uh oh!

kumargauravin commented Aug 10, 2020 via email

Uh oh!

kumargauravin commented Aug 23, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stale bot commented Oct 12, 2020

Uh oh!

feluelle commented Oct 22, 2020

Uh oh!

kumargauravin commented Oct 22, 2020 via email

Uh oh!

feluelle commented Oct 28, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kumargauravin commented Oct 29, 2020 via email

Uh oh!

stale bot commented Dec 25, 2020

Uh oh!

malthe commented Dec 16, 2021

Uh oh!

kumargauravin commented Dec 16, 2021 via email

Uh oh!

kumargauravin commented Dec 16, 2021 via email

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ephraimbuddy Aug 10, 2020 •

edited

Loading

feluelle commented Oct 28, 2020 •

edited

Loading