spatial and temporal distribution of service calls using big data tools

[priyankatuteja]

<insert pull request description here>

---

Checklist

Please go through each entry in the below checklist and mark an 'X' if that condition has been met. Every entry should be marked with an 'X' to be get the Pull Request approved.

• All imports are in the first cell? First block of imports are standard libraries, second block are 3rd party libraries, third block are all arcgis imports?
• All GIS object instantiations are one of the following?
  • gis = GIS('https://www.arcgis.com', 'arcgis_python', 'P@ssword123')
  • gis = GIS(profile="your_online_profile")
  • gis = GIS('https://pythonapi.playground.esri.com/portal', 'arcgis_python', 'amazing_arcgis_123')
  • gis = GIS(profile="your_enterprise_portal")
• If this notebook requires setup or teardown, did you add the appropriate code to ./misc/setup.py and/or ./misc/teardown.py?
• [] If this notebook references any portal items that need to be staged on AGOL/Python API playground, did you coordinate with a Python API team member to stage the item the correct way with the api_data_owner user?
• Code refactored & split out across multiple cells, useful comments?
• Consistent voice/tense/narrative style? Thoroughly checked for typos?
• All images used like <img src="base64str_here"> instead of <img src="https://some.url">? All map widgets contain a static image preview? (Call mapview_inst.take_screenshot() to do so)
• All file paths are constructed in an OS-agnostic fashion with os.path.join()? (Instead of r"\foo\bar", os.path.join(os.path.sep, "foo", "bar"), etc.)
• [] IF YOU WANT THIS SAMPLE TO BE DISPLAYED ON THE DEVELOPERS.ARCGIS.COM WEBSITE, ping @ DavidJVitale so he can add it to the list for the next deploy

[review-notebook-app]

Check out this pull request on ReviewNB: https://app.reviewnb.com/Esri/arcgis-python-api/pull/497

You'll be able to see notebook diffs and discuss changes. Powered by ReviewNB.

[AtmaMani]

@priyankatuteja are you waiting to get BDFS configured on PythonAPI playground portal for this PR?

[priyankatuteja]

@priyankatuteja are you waiting to get BDFS configured on PythonAPI playground portal for this PR?

@AtmaMani I am waiting to get the sample reviewed and for that, I assume you need data to test the notebook. Please let me know where to add BDFS for this sample so this could be reviewed and approved accordingly

[priyankatuteja]

@priyankatuteja are you waiting to get BDFS configured on PythonAPI playground portal for this PR?

@AtmaMani I am waiting to get the sample reviewed and for that, I assume you need data to test the notebook. Please let me know where to add BDFS for this sample so this could be reviewed and approved accordingly

[AtmaMani]

@priyankatuteja yes please follow up on the email thread you started to get the data published and update the connection strings accordingly. Meanwhile I will review the content

[AtmaMani]

@jyaistMap could you do the first round of reviews for this PR?

[jyaistMap]

Hi @priyankatuteja I'm reviewing the sample, but do not have access to \\DELDEVAL031\datastore. Are you able to access \\JYAIST2\sample_review? Can you place the data there?

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB (backstory for this conversation format).

jyaistMap commented on 2019-10-04T16:57:19Z
----------------------------------------------------------------

I've made a ton of suggestions about adding some explanatory text. I understand this is a sample rather than a guide. It might help clarify at the beginning of this sample what we expect the audience to already know.

• the process_spatial_reference code confused me with no explanation
• the run_python_code geoanalytics tool also confused me without any knowledge of output formats

I think links to relevant help at the beginning of the sample could benefit users:

• Run Python Script
• Using Geonalytics Tasks in Run Python Script

priyankatuteja commented on 2019-11-09T12:34:29Z
----------------------------------------------------------------

will it be a good idea to have a further reading section wherein I could include hyperlinks like you suggested above. The idea is to keep consistency with other samples. This allows a user/reader to skim and scan the sample in order to understand the main idea.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB (backstory for this conversation format).

jyaistMap commented on 2019-10-04T16:57:20Z
----------------------------------------------------------------

Given the wording is "sample aims to find answers to some fundamental questions:", I'd reword the bullet points to the form of questions:

• What is the spatial relationship between 911 calls?
• Which block groups have the highest number of 911 calls reporting?
• What is the most common reason for 911 calls?
• How many 911 calls occur each month?
• How many 911 calls occur each hour?

I suggest telling people how to find the specific data at data.gov. I typed a search for new orleans 911 incidents and it looks like that's the data. It might help to point people there to get additional information about interpreting the results.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB (backstory for this conversation format).

jyaistMap commented on 2019-10-04T16:57:21Z
----------------------------------------------------------------

• remove the redundant datetime import on line 5

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB (backstory for this conversation format).

jyaistMap commented on 2019-10-04T16:57:22Z
----------------------------------------------------------------

Change the first word to an active verb to stay consistent:

Ensure...

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB (backstory for this conversation format).

jyaistMap commented on 2019-10-04T16:57:22Z
----------------------------------------------------------------

Might help to clarify that you've set up a directory and file structure described above at a specific location.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB (backstory for this conversation format).

jyaistMap commented on 2019-10-04T16:57:23Z
----------------------------------------------------------------

I suggest changing the sentence and linking to the manifest documenation:

The Geoanalytics server samples the datasets in the big data file share to generate a manifest, which outlines the data schema and specifies any time and geometry fields. A query of the resulting manifest returns each dataset's schema.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB (backstory for this conversation format).

jyaistMap commented on 2019-10-04T16:57:24Z
----------------------------------------------------------------

Maybe clarify a bit about the spatial reference:

but we know this data is from New Orleans, Louisiana, and is actually stored in the Louisiana State Plane Coordinate System. We need to edit the manifest with the correct spatial reference: {"wkid": 102682, "latestWkid": 3452}

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB (backstory for this conversation format).

jyaistMap commented on 2019-10-04T16:57:25Z
----------------------------------------------------------------

Use an active verb to stay consistent:

Search...

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB (backstory for this conversation format).

jyaistMap commented on 2019-10-04T16:57:25Z
----------------------------------------------------------------

I suggest linking to the documentation to clarify terms as they are presented in a sample:

Adding a big data file share to the Geoanalytics server adds a corresponding big data file share item on the portal. We can search for these types of items using the item_type parameter.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB (backstory for this conversation format).

jyaistMap commented on 2019-10-04T16:57:26Z
----------------------------------------------------------------

I would add a sentence describing what you are doing here. I'd also clarify the type of Python API object this returns. It looks like it would return a Feature Layer, but the properties property in the cell below looks very different from the output of Feature Layer.properties :

Querying the layers property of the item returns a <API Object Type> representing the data. The object is actually an API Layer object.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB (backstory for this conversation format).

jyaistMap commented on 2019-10-04T16:57:27Z
----------------------------------------------------------------

Stay consistent:

Search...

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB (backstory for this conversation format).

jyaistMap commented on 2019-10-04T16:57:28Z
----------------------------------------------------------------

I'd delete this sentence or remove the bold.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB (backstory for this conversation format).

jyaistMap commented on 2019-10-04T16:57:30Z
----------------------------------------------------------------

I'd suggest hyperlinks for the different objects:

The describe_dataset provides an overview of the big data file share item layer. By default, the tool outputs a table...

Optionally, the tool can output a feature layer representing a sample set of features using the sample_size parameter, or a single polygon feature layer representing the input feature layers' extent by setting the extent_output parameter to True.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB (backstory for this conversation format).

jyaistMap commented on 2019-10-04T16:57:30Z
----------------------------------------------------------------

The describe_dataset documentation explicitly states that the tool outputs a Feature Layer. I found in testing it output arcgis.gis.Item, so to get the feature layer you'd need to get the layers return values.

Consequently, I received errors on the output_json and sample_layer property returns in the next cells:

---------------------------------------------------------------------------

AttributeError: 'Item' object has no attribute 'ouput_json'

I don't know what output_json is supposed to return, but I could not retrieve it. description.get_data() returned nothing as well.

I found the output description item had 2 layers, one named extent_layer and one named sample_layer. So instead I could get retrieve the results through this code:

sample_layer = description.layers[1]

priyankatuteja commented on 2019-11-10T02:33:46Z
----------------------------------------------------------------

I need to test this tool to confirm whether the output type has been changed for some reason.

priyankatuteja commented on 2019-11-11T10:45:09Z
----------------------------------------------------------------

This solves problem with return_tuple=True

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB (backstory for this conversation format).

jyaistMap commented on 2019-10-04T16:57:31Z
----------------------------------------------------------------

I suggest:

We can see some records have missing or invalid attribute values, including in the fields the manifests defines as the time and geometry values.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB (backstory for this conversation format).

jyaistMap commented on 2019-10-04T16:57:32Z
----------------------------------------------------------------

typo:

New Orlans....should be New Orleans

beacause....should be because

Change second sentence:

We want to explore data points within New Orleans city limits.

Change third sentence and include a hyperlink:

We will use the clip_layer tool to extract only point features within the New Orleans boundary.

Remove the last sentence, it's redundant.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB (backstory for this conversation format).

jyaistMap commented on 2019-10-04T16:57:32Z
----------------------------------------------------------------

I'd move this cell above the paragraph above so the user can see what you're describing.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB (backstory for this conversation format).

jyaistMap commented on 2019-10-04T16:57:33Z
----------------------------------------------------------------

I'd rephrase:

Extract features within New Orleans

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB (backstory for this conversation format).

jyaistMap commented on 2019-10-04T16:57:34Z
----------------------------------------------------------------

Overlay seems like the wrong word.

The clip_layer method extracts input point, line, or polygon features from an input_layer that fall within the boundaries of features in a clip_layer. The output layer contains a subset of features from the input layer. We will clip our input call feature layer to the New Orleans blk_grp_lyr features.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB (backstory for this conversation format).

jyaistMap commented on 2019-10-04T16:57:34Z
----------------------------------------------------------------

Do we know why there is no thumbnail for the clip_result?

priyankatuteja commented on 2019-11-10T02:54:29Z
----------------------------------------------------------------

Need to find out the reason.

priyankatuteja commented on 2019-11-12T06:10:25Z
----------------------------------------------------------------

Looks like it has been there for all tools, see https://developers.arcgis.com/python/sample-notebooks/analyzing-violent-crime/

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB (backstory for this conversation format).

jyaistMap commented on 2019-10-04T16:57:35Z
----------------------------------------------------------------

• a more descriptive variable might be orleans_calls
• I suggest adding a cell to visualize the orleans_call features on a map.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB (backstory for this conversation format).

jyaistMap commented on 2019-10-04T16:57:42Z
----------------------------------------------------------------

Do we know why the thumbnail is missing?

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB (backstory for this conversation format).

jyaistMap commented on 2019-10-04T16:57:43Z
----------------------------------------------------------------

I would add a sentence or two explaining what the data says:

The darkest red features indicate areas where you can state with 99 percent confidence that the clustering of 911 call features is not the result of random chance but rather of some other variable that might be worth investigating. Similarly, the darkest blue features indicate that the lack of 911 calls is most likely not just random, but with 90% certainty you can state it is because of some variable in those locations. Features that are beige do not represent statistically significant clustering; the number of 911 calls could very likely be the result of random processes and random chance in those areas.

It might also help to add the Block Group layer over top of the hot spots and symbolize with no color but just a thicker outline that would allow users to zoom in and investigate which block groups have clustering and which don't.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB (backstory for this conversation format).

jyaistMap commented on 2019-10-04T16:57:44Z
----------------------------------------------------------------

I would change the heading to use an active very and simplify to remain consistent with other headings

Visualize other aspects

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB (backstory for this conversation format).

jyaistMap commented on 2019-10-04T16:57:45Z
----------------------------------------------------------------

Some suggested edits:

The run_python_script method executes a Python script directly in an ArcGIS GeoAnalytics server site . The script can create an analysis pipeline by chaining together multiple GeoAnalytics Tools without writing intermediate results to a data store. The tool can also distribute Python functionality across the GeoAnalytics server site.

Geoanalytics Server installs a Python 3.6 environment that this tool uses. The environment includes Spark 2.2.0, the compute platform that distributes analysis across multiple cores of one or more machines in your GeoAnalytics Server site. The environment includes the pyspark module which provides a collection of distributed analysis tools for data management, clustering, regression, and more. The run_python_script task automatically imports the pyspark module so you can directly interact with it.

When using the geoanalytics and pyspark packages, most functions return analysis results as Spark DataFrame memory structures. You can write these data frames to a data store or process them in a script. This lets you chain multiple geoanalytics and pyspark tools while only writing out the final result, eliminating the need to create any bulky intermediate result layers.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB (backstory for this conversation format).

jyaistMap commented on 2019-10-04T16:57:45Z
----------------------------------------------------------------

Reorder the sentence

The TextType field contains the reason for the initial 911 call. Let's investigate the most frequent purposes for 911 calls in the New Orleans area by writing our own function:

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB (backstory for this conversation format).

jyaistMap commented on 2019-10-04T16:57:46Z
----------------------------------------------------------------

• remove the datetime import since we did that in the Necessary Imports cell at the beginning of the sample

• I'm confused by lines 3 and 4. The calls variable was defined earlier with this statement: calls = data_item.layers[0]

The data_item is a Big Data File Share item instance. Does the layers property of a big data file share item create a different type of layer than a Feature Layer? It's quite possible I don't understand how the run_python_code tool works, but I'm not understanding how the function above creates a pandas dataframe. The layers parameter in run_python_script is described in the documentation as a list of Feature Layers. The groupby_texttype() function above is creating a dataframe in lines 3 and 4 with df = layers[0], but layers is a list of feature layers, and returning the first feature layer with [0] just means df is a Feature Layer not a dataframe. Am I missing something? Wouldn't line 4 need to be df = layers[0].query(where="1=1", as_df=True)?

I'm assuming I'm wrong, but perhaps explaining what's happening in the function and how it relates to the run_python_script input would help clarify.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB (backstory for this conversation format).

jyaistMap commented on 2019-10-04T16:57:46Z
----------------------------------------------------------------

I suggest being explicit in indicating that the run_python_code outputs an item on the portal controlled by the out.write.format line. Is there documentation on how to do this?

I also suggest explaining how to find the output on the portal by reading the message output of the tool

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB (backstory for this conversation format).

jyaistMap commented on 2019-10-04T16:57:47Z
----------------------------------------------------------------

I would rephrase the above sentence:

Now we'll investigate 911 calls to investigate the most common reasons for calls within block addresses. We will define a new function to input to the run_python_script tool, this time grouping data by the TextType and BLOCK_ADDRESS attributes.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB (backstory for this conversation format).

jyaistMap commented on 2019-10-04T16:57:48Z
----------------------------------------------------------------

I would add explanatory text for the lines within the function.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB (backstory for this conversation format).

jyaistMap commented on 2019-10-04T16:57:49Z
----------------------------------------------------------------

Rephrase:

The result indicates the most common reason for a 911 call in the 22A Block in New Orleans is defined as AREA CHECK.

It might also help to tell people they can get more specific information about this data and what these attributes mean by looking for the data themselves and exploring the metatada:

Go to data.gov and search for new orleans 911 incidents to explore the metadata for this dataset.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB (backstory for this conversation format).

jyaistMap commented on 2019-10-04T16:57:49Z
----------------------------------------------------------------

I would be more explicit in explaining what's coming up:

Now let's investigate the 911 call data for temporal trends. We saw in the manifest that the TimeCreate field holds specific time information for one instant (in UTC time zone) using a string format: MM/dd/yyyy hh:mm:ss a. We can parse these strings using Python's datetime` module to extract year, month, day, hour, minute, and second to perform time analyses.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB (backstory for this conversation format).

jyaistMap commented on 2019-10-04T16:57:50Z
----------------------------------------------------------------

Let's define a helper function to convert the TimeCreate attribute field string types into a date type.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB (backstory for this conversation format).

jyaistMap commented on 2019-10-04T16:57:51Z
----------------------------------------------------------------

• remove the datetime import as it is imported in the Necessary Imports section
• provide a link to the pyspark.sql module help
• perhaps this is not the right place to do it, but as mentioned above, I'm confused as to what df actually is. I'd suggest explaining the types of input and output. If that's more appropriate for a guide, or is fundamental knowledge data scientists possess, I understand that decision.
• It would help to explain in more detail, or reiterating that we're using this because it distributes the analysis. Why use the pyspark.sql functions instead of using datetime.strptime('TimeCreate', %m/%d/%Y %H:%M:%S).? Is it because of the size of this dataset? Is the number of records? Why is distributed analysis so crucial for this data?

priyankatuteja commented on 2019-11-11T05:38:14Z
----------------------------------------------------------------

since the run_python_script tool is executed directly on GA server site. The imports made in notebook do not work for code within script

priyankatuteja commented on 2019-11-11T05:44:24Z
----------------------------------------------------------------

The data is too big to use datetime.strptime('TimeCreate', %m/%d/%Y %H:%M:%S). We need distributed platform to perform this.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB (backstory for this conversation format).

jyaistMap commented on 2019-10-04T16:57:51Z
----------------------------------------------------------------

typo:

eralier should be earlier

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB (backstory for this conversation format).

jyaistMap commented on 2019-10-04T16:57:52Z
----------------------------------------------------------------

I believe the functions written as input to the run_python_code deserve a bit more explanation as to what is happening. I think we should write these samples with the perspective of beginners in mind, but I also understand we're not providing a pandas and/or pyspark tutorial.

I found myself sidetracked and looking up documentation for how the functions were supposed to work. I also have NO experience with data science workflows and distributed analysis with pandas or pyspark, so if we conclude that the audience for this sample will already know this type of information, you can ignore many of my comments :)

priyankatuteja commented on 2019-11-11T15:29:58Z
----------------------------------------------------------------

I have been trying to add some hyperlinks so our users can quickly look up for more details. It really doesn't seem like you have NO experience with such workflows. These suggestions are really meaningful.Thanks a lot!

[jyaistMap]

@AtmaMani @priyankatuteja
I set up a big data file share on the pythonapi playground using data I retrieved after searching for new orleans service calls - see the link to the search results here. I tested the notebook against that big data files share item, and am making some additional notes.

[priyankatuteja]

@jyaistMap Thanks for the suggestions. I was occupied with python api testing and documentation. Apologies for the delayed response. I will be incorporating these suggestions soon. Also placing the data on \JYAIST2\sample_review for further testing of the sample.

[priyankatuteja]

@jyaistMap I do not have access to \JYAIST2\sample_review. But, I have added you and Atma to have access to \DELDEVAL031\datastore. Please let me know if that still does not work.

[priyankatuteja]

@AtmaMani, @jyaistMap I have made changes to the notebook. Please let me know if any other changes required.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB (backstory for this conversation format).

AtmaMani commented on 2019-11-14T22:49:53Z
----------------------------------------------------------------

"You can obtain data by searching using keywords, for exxample" - typo in word example

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB (backstory for this conversation format).

AtmaMani commented on 2019-11-14T22:49:54Z
----------------------------------------------------------------

In this 2nd aggregation that runs on pySpark, you are grouping by both BLOCK_ADDRESS and TextType. Hence, the result has counts split by address group for each text type. Thus, I think when you are plotting the DataFrame, by just address, you need to run another groupby on just the BLOCK_ADDRESS otherwise your counts may not be correct. Is that the case, could you check?

priyankatuteja commented on 2019-11-15T16:46:55Z
----------------------------------------------------------------

I confirmed it, that is not the case. This groupby gives correct count.

[AtmaMani]

@priyankatuteja the latest revision looks much better. This is a really good sample. I think we can borrow from this when we rewrite the geoanalytics module guide. I have asked a question about one of the aggregation methods, could you take a look at that?

[priyankatuteja]

@AtmaMani Thanks for the review! I have made the changes you suggested.

[AtmaMani]

Changes look good

[AtmaMani]

@priyankatuteja thanks for the sample, @jyaistMap thanks for the thorough review