Adding official version for Fraud detection notebook #321

sudarshan-SpringML · 2022-02-23T12:13:58Z

This PR contains a notebook which shows you how to build, deploy, and analyze predictions from a simple random forest model using tools like scikit-learn, Vertex AI, and the What-IF Tool (WIT) on a synthetic fraud transaction dataset to solve a financial fraud detection problem.

Use the notebook template as a starting point.
Follow the style and grammar rules outlined in the above notebook template.
Verify the notebook runs successfully in Colab since the automated tests cannot guarantee this even when it passes.
Passes all the required automated checks. You can locally test for formatting and linting with these instructions.
You have consulted with a tech writer to see if tech writer review is necessary. If so, the notebook has been reviewed by a tech writer, and they have approved it.
This notebook has been added to the CODEOWNERS file under # Official Notebooks section, pointing to the author or the author's team.
The Jupyter notebook cleans up any artifacts it has created (datasets, ML models, endpoints, etc) so as not to eat up unnecessary resources.

review-notebook-app · 2022-02-23T12:14:02Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

kweinmeister · 2022-02-23T19:21:13Z

/gcbrun

andrewferlitsch · 2022-02-25T00:52:57Z

Build error:
ModuleNotFoundError Traceback (most recent call last)
Step #3: Input In [17], in
Step #3: 1 import warnings
Step #3: ----> 3 import joblib
Step #3: 4 import matplotlib.pyplot as plt
Step #3: 5 import numpy as np
Step #3:
Step #3: ModuleNotFoundError: No module named 'joblib'

ivanmkc · 2022-02-25T16:36:18Z

/gcbrun

ivanmkc · 2022-02-25T17:05:42Z

tldr: pip install joblib. You don't need an IS_TESTING check as this is a required dependency. Don't worry about what Workbench or Colab have preinstalled, just install any dependencies that our base image doesn't provide. That is the source-of-truth.

…an-SpringML/vertex-ai-samples into fraud_detection_notebook

sudarshan-SpringML · 2022-03-04T15:06:39Z

notebook ready for review

kweinmeister · 2022-03-04T16:33:41Z

/gcbrun

kweinmeister · 2022-03-04T16:37:02Z

notebooks/official/workbench/fraud_detection/fraud-detection-model.ipynb

@@ -9,6 +9,57 @@
        "# Build a fraud detection model on Vertex AI"


Line #8. serving_container_image_uri="us-docker.pkg.dev/vertex-ai/prediction/sklearn-cpu.0-24:latest",
Can we pull this out as a constant variable? Also, let's use the latest version 1.0 listed here: https://cloud.google.com/vertex-ai/docs/predictions/pre-built-containers

Reply via ReviewNB

kweinmeister · 2022-03-04T16:38:55Z

@sudarshan-SpringML added one comment here: https://app.reviewnb.com/GoogleCloudPlatform/vertex-ai-samples/pull/321/

Other than that, looks good!

kweinmeister · 2022-03-07T15:58:49Z

/gcbrun

sudarshan-SpringML · 2022-03-11T15:28:44Z

Thanks Karl. Will check that

…

On Fri, Mar 11, 2022 at 8:44 PM Karl Weinmeister ***@***.***> wrote: I believe yes, markdown cells count, but not 100% sure. I'd check both cells counting markdown and not. — Reply to this email directly, view it on GitHub <#321 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATV6CWCGH2GEE3L2YGN7CP3U7NPMTANCNFSM5PEFPCTQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Sudarshan C | 901 076 8745 Associate Data Engineer | SpringML <https://springml.com/?utm_source=WiseStamp&utm_medium=email&utm_term&utm_content&utm_campaign=signature> <https://www.springml.com/events/webinar-series-building-modern-data-stack/?utm_source=email_signature&utm_medium=xink&utm_campaign=Webinar_DataStackSeries> Video: SpringML’s cutting-edge Floor Visualizer Solution <https://www.springml.com/videos/springmls-cutting-edge-floor-visualizer-solution/>

sudarshan-SpringML · 2022-03-14T15:37:40Z

@sudarshan-SpringML the kernel died again. Can you please check the specific cell below? Note that cell 66 is the one that prints out the dimensions. Two cells below, the issue occurs.

INFO:papermill:Executing Cell 66--------------------------------------
INFO:papermill:(4453834, 11) (1908786, 11)

(4453834, 11) (1908786, 11)
INFO:papermill:Ending Cell 66-----------------------------------------
INFO:papermill:Executing Cell 67--------------------------------------
INFO:papermill:Ending Cell 67-----------------------------------------
INFO:papermill:Executing Cell 68--------------------------------------
ERROR:papermill:Kernel died while waiting for execute reply.
INFO:papermill:Ending Cell 68-----------------------------------------

@kweinmeister I think this cell is failing for me-> "forest = RandomForestClassifier()
forest.fit(X_train, y_train)". For me this cell takes around 10-15 mins.

sudarshan-SpringML · 2022-03-22T06:55:14Z

added debug statements to find the cell where kernel dies

andrewferlitsch · 2022-03-24T18:52:25Z

/gcbrun

kweinmeister · 2022-03-24T19:04:57Z

/gcbrun

andrewferlitsch · 2022-03-25T20:31:23Z

Weird error. See if I can force a retest.

Traceback (most recent call last):
Step #3: File "/workspace/.cloud-build/execute_notebook_cli.py", line 36, in
Step #3: execute_notebook_helper.execute_notebook(
Step #3: File "/workspace/.cloud-build/execute_notebook_helper.py", line 91, in execute_notebook
Step #3: raise execution_exception
Step #3: File "/workspace/.cloud-build/execute_notebook_helper.py", line 56, in execute_notebook
Step #3: pm.execute_notebook(
Step #3: File "/builder/home/.local/lib/python3.9/site-packages/papermill/execute.py", line 107, in execute_notebook
Step #3: nb = papermill_engines.execute_notebook_with_engine(
Step #3: File "/builder/home/.local/lib/python3.9/site-packages/papermill/engines.py", line 49, in execute_notebook_with_engine
Step #3: return self.get_engine(engine_name).execute_notebook(nb, kernel_name, **kwargs)
Step #3: File "/builder/home/.local/lib/python3.9/site-packages/papermill/engines.py", line 359, in execute_notebook
Step #3: cls.execute_managed_notebook(nb_man, kernel_name, log_output=log_output, **kwargs)
Step #3: File "/builder/home/.local/lib/python3.9/site-packages/papermill/engines.py", line 418, in execute_managed_notebook
Step #3: return PapermillNotebookClient(nb_man, **final_kwargs).execute()
Step #3: File "/builder/home/.local/lib/python3.9/site-packages/papermill/clientwrap.py", line 45, in execute
Step #3: self.papermill_execute_cells()
Step #3: File "/builder/home/.local/lib/python3.9/site-packages/papermill/clientwrap.py", line 72, in papermill_execute_cells
Step #3: self.execute_cell(cell, index)
Step #3: File "/builder/home/.local/lib/python3.9/site-packages/nbclient/util.py", line 84, in wrapped
Step #3: return just_run(coro(*args, **kwargs))
Step #3: File "/builder/home/.local/lib/python3.9/site-packages/nbclient/util.py", line 62, in just_run
Step #3: return loop.run_until_complete(coro)
Step #3: File "/usr/local/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
Step #3: return future.result()
Step #3: File "/builder/home/.local/lib/python3.9/site-packages/nbclient/client.py", line 953, in async_execute_cell
Step #3: raise DeadKernelError("Kernel died")
Step #3: nbclient.exceptions.DeadKernelError: Kernel died

andrewferlitsch · 2022-03-31T23:47:45Z

/gcbrun

andrewferlitsch · 2022-04-01T01:34:56Z

Step #3: Traceback (most recent call last):
Step #3: File "/workspace/.cloud-build/execute_notebook_cli.py", line 36, in
Step #3: execute_notebook_helper.execute_notebook(
Step #3: File "/workspace/.cloud-build/execute_notebook_helper.py", line 91, in execute_notebook
Step #3: raise execution_exception
Step #3: File "/workspace/.cloud-build/execute_notebook_helper.py", line 56, in execute_notebook
Step #3: pm.execute_notebook(
Step #3: File "/builder/home/.local/lib/python3.9/site-packages/papermill/execute.py", line 107, in execute_notebook
Step #3: nb = papermill_engines.execute_notebook_with_engine(
Step #3: File "/builder/home/.local/lib/python3.9/site-packages/papermill/engines.py", line 49, in execute_notebook_with_engine
Step #3: return self.get_engine(engine_name).execute_notebook(nb, kernel_name, **kwargs)
Step #3: File "/builder/home/.local/lib/python3.9/site-packages/papermill/engines.py", line 359, in execute_notebook
Step #3: cls.execute_managed_notebook(nb_man, kernel_name, log_output=log_output, **kwargs)
Step #3: File "/builder/home/.local/lib/python3.9/site-packages/papermill/engines.py", line 418, in execute_managed_notebook
Step #3: return PapermillNotebookClient(nb_man, **final_kwargs).execute()
Step #3: File "/builder/home/.local/lib/python3.9/site-packages/papermill/clientwrap.py", line 45, in execute
Step #3: self.papermill_execute_cells()
Step #3: File "/builder/home/.local/lib/python3.9/site-packages/papermill/clientwrap.py", line 72, in papermill_execute_cells
Step #3: self.execute_cell(cell, index)
Step #3: File "/builder/home/.local/lib/python3.9/site-packages/nbclient/util.py", line 84, in wrapped
Step #3: return just_run(coro(*args, **kwargs))
Step #3: File "/builder/home/.local/lib/python3.9/site-packages/nbclient/util.py", line 62, in just_run
Step #3: return loop.run_until_complete(coro)
Step #3: File "/usr/local/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
Step #3: return future.result()
Step #3: File "/builder/home/.local/lib/python3.9/site-packages/nbclient/client.py", line 953, in async_execute_cell
Step #3: raise DeadKernelError("Kernel died")
Step #3: nbclient.exceptions.DeadKernelError: Kernel died
Finished Step #3

andrewferlitsch · 2022-04-04T16:46:33Z

@kweinmeister Has anyone debugged the reason for kernel died?

kweinmeister · 2022-04-05T17:52:48Z

/gcbrun

andrewferlitsch · 2022-04-12T17:03:27Z

kernel died again

ivanmkc · 2022-04-19T10:14:45Z

/gcbrun

ivanmkc · 2022-04-19T11:57:58Z

Seems like it's dying on forest.fit(X_train, y_train).

Try initializing RandomForestClassifier with verbose=1 to get more info.

Reference: https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html

sudarshan-SpringML · 2022-04-20T06:30:06Z

Seems like it's dying on forest.fit(X_train, y_train).

Try initializing RandomForestClassifier with verbose=1 to get more info.

Reference: https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html

Done

andrewferlitsch · 2022-04-21T02:24:37Z

@sudarshan-SpringML Please run this notebook manually in Workbench AI and see if the kernel dies or not.

sudarshan-SpringML · 2022-04-21T03:53:09Z

Hi @andrew Ferlitsch ***@***.***>. I ran notebook in workbench yesterday. It worked fine

…

On Thu, Apr 21, 2022, 7:54 AM Andrew Ferlitsch ***@***.***> wrote: @sudarshan-SpringML <https://github.com/sudarshan-SpringML> Please run this notebook manually in Workbench AI and see if the kernel dies or not. — Reply to this email directly, view it on GitHub <#321 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATV6CWG75ZZUAOTTK7J7P7DVGC37BANCNFSM5PEFPCTQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

andrewferlitsch · 2022-04-22T05:07:46Z

per comment, notebook runs a-ok in Workbench, so will manually merge.

sudarshan-SpringML added 3 commits February 23, 2022 12:07

deleted file in community folder

867b4dd

modified notebook

c0d60cf

ran linter test

7ede786

sudarshan-SpringML requested a review from a team as a code owner February 23, 2022 12:13

Merge branch 'main' into fraud_detection_notebook

3f794e3

Merge branch 'main' into fraud_detection_notebook

e68aa49

sudarshan-SpringML added 2 commits February 28, 2022 06:17

renamed managed_notebooks folder to workbench

82b3aa8

ran linter

9c471ab

sudarshan-SpringML changed the title ~~Fraud detection notebook~~ Adding official version for Fraud detection notebook Mar 1, 2022

kweinmeister and others added 6 commits March 2, 2022 09:02

Merge branch 'main' into fraud_detection_notebook

b519bf1

resolved comments

dd46198

ran linter test

98dfeaf

Merge branch 'fraud_detection_notebook' of https://github.com/sudarsh…

8abc357

…an-SpringML/vertex-ai-samples into fraud_detection_notebook

pulled new version of branch

8616409

ran linter again

d1a04ea

Merge branch 'main' into fraud_detection_notebook

7d73bc8

kweinmeister reviewed Mar 4, 2022

View reviewed changes

sudarshan-SpringML and others added 3 commits March 7, 2022 09:47

resolved comments

d817e1f

ran linter test

c338314

Merge branch 'main' into fraud_detection_notebook

3f785b7

Merge branch 'main' into fraud_detection_notebook

99519f3

sudarshan-SpringML added 2 commits March 22, 2022 06:52

added debug statements

9af893b

ran linter test

52eeb29

Merge branch 'main' into fraud_detection_notebook

1e5b6fb

andrewferlitsch added 3 commits March 25, 2022 13:31

Merge branch 'main' into fraud_detection_notebook

5e32bad

Merge branch 'main' into fraud_detection_notebook

93cf107

Merge branch 'main' into fraud_detection_notebook

14f0681

andrewferlitsch and others added 2 commits April 4, 2022 09:46

Merge branch 'main' into fraud_detection_notebook

d87e6e5

Merge branch 'main' into fraud_detection_notebook

015057c

Merge branch 'main' into fraud_detection_notebook

313ba55

sudarshan-SpringML added 2 commits April 20, 2022 06:27

added verbose

9783237

ran linter test

5680783

Merge branch 'main' into fraud_detection_notebook

fa5620f

andrewferlitsch merged commit 831c515 into GoogleCloudPlatform:main Apr 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding official version for Fraud detection notebook #321

Adding official version for Fraud detection notebook #321

sudarshan-SpringML commented Feb 23, 2022 •

edited

review-notebook-app bot commented Feb 23, 2022

kweinmeister commented Feb 23, 2022

andrewferlitsch commented Feb 25, 2022

ivanmkc commented Feb 25, 2022

ivanmkc commented Feb 25, 2022

sudarshan-SpringML commented Mar 4, 2022

kweinmeister commented Mar 4, 2022

kweinmeister Mar 4, 2022

kweinmeister commented Mar 4, 2022

kweinmeister commented Mar 7, 2022

sudarshan-SpringML commented Mar 11, 2022 via email

sudarshan-SpringML commented Mar 14, 2022

sudarshan-SpringML commented Mar 22, 2022

andrewferlitsch commented Mar 24, 2022

kweinmeister commented Mar 24, 2022

andrewferlitsch commented Mar 25, 2022

andrewferlitsch commented Mar 31, 2022

andrewferlitsch commented Apr 1, 2022

andrewferlitsch commented Apr 4, 2022

kweinmeister commented Apr 5, 2022

andrewferlitsch commented Apr 12, 2022

ivanmkc commented Apr 19, 2022

ivanmkc commented Apr 19, 2022

sudarshan-SpringML commented Apr 20, 2022

andrewferlitsch commented Apr 21, 2022

sudarshan-SpringML commented Apr 21, 2022 via email

andrewferlitsch commented Apr 22, 2022

		@@ -9,6 +9,57 @@
		"# Build a fraud detection model on Vertex AI"

Adding official version for Fraud detection notebook #321

Adding official version for Fraud detection notebook #321

Conversation

sudarshan-SpringML commented Feb 23, 2022 • edited

review-notebook-app bot commented Feb 23, 2022

kweinmeister commented Feb 23, 2022

andrewferlitsch commented Feb 25, 2022

ivanmkc commented Feb 25, 2022

ivanmkc commented Feb 25, 2022

sudarshan-SpringML commented Mar 4, 2022

kweinmeister commented Mar 4, 2022

kweinmeister Mar 4, 2022

Choose a reason for hiding this comment

kweinmeister commented Mar 4, 2022

kweinmeister commented Mar 7, 2022

sudarshan-SpringML commented Mar 11, 2022 via email

sudarshan-SpringML commented Mar 14, 2022

sudarshan-SpringML commented Mar 22, 2022

andrewferlitsch commented Mar 24, 2022

kweinmeister commented Mar 24, 2022

andrewferlitsch commented Mar 25, 2022

andrewferlitsch commented Mar 31, 2022

andrewferlitsch commented Apr 1, 2022

andrewferlitsch commented Apr 4, 2022

kweinmeister commented Apr 5, 2022

andrewferlitsch commented Apr 12, 2022

ivanmkc commented Apr 19, 2022

ivanmkc commented Apr 19, 2022

sudarshan-SpringML commented Apr 20, 2022

andrewferlitsch commented Apr 21, 2022

sudarshan-SpringML commented Apr 21, 2022 via email

andrewferlitsch commented Apr 22, 2022

sudarshan-SpringML commented Feb 23, 2022 •

edited