[SPARK-38484][PYTHON] Move usage logging instrumentation util functions from pandas module to pyspark.util module

[heyihong]

What changes were proposed in this pull request?

Move usage logging instrumentation util functions from pandas module to pyspark.util module

Why are the changes needed?

It will be helpful to attach the usage logger to other modules (e.g. sql) besides Pandas but other modules should not depend on Pandas modules to use the instrumentation utils (e.g. _wrap_function, _wrap_property ...).

Does this PR introduce any user-facing change?

No

How was this patch tested?

• Existing unit tests
• Manual test by running ./bin/pyspark and verified the output:

>>> sc.setLogLevel("info")
>>> import pyspark.pandas as ps
22/03/15 17:17:16 INFO Log4jUsageLogger: pandasOnSparkImported=1.0, tags=List(), blob=
22/03/15 17:18:21 INFO Log4jUsageLogger: initialConfigLogging=1.0, tags=List(sparkApplicationId=local-1647360920525, sparkExecutionId=null, sparkJobGroupId=null), blob={"spark.sql.warehouse.dir":"file:/Users/yihong.he/spark/spark-warehouse","spark.executor.extraJavaOptions":"-XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED","spark.driver.host":"10.120.131.148","spark.serializer.objectStreamReset":"100","spark.driver.port":"61338","spark.rdd.compress":"True","spark.app.name":"PySparkShell","spark.submit.pyFiles":"","spark.ui.showConsoleProgress":"true","spark.app.startTime":"1647360919721","spark.executor.id":"driver","spark.driver.extraJavaOptions":"-XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED","spark.submit.deployMode":"client","spark.master":"local[*]","spark.sql.catalogImplementation":"hive","spark.app.id":"local-1647360920525"}
22/03/15 17:18:25 INFO Log4jUsageLogger: pandasOnSparkFunctionCalled=1.0, tags=List(pandasOnSparkFunction=__init__(self, data=None, index=None, columns=None, dtype=None, copy=False), className=DataFrame, status=success), blob={"duration": 3781.477633999998}
>>> psdf.columns
22/03/15 17:18:49 INFO Log4jUsageLogger: pandasOnSparkFunctionCalled=1.0, tags=List(pandasOnSparkProperty=columns, className=DataFrame, status=success), blob={"duration": 0.24742499999774736}
Index(['a', 'b'], dtype='object')

[hvanhovell]

Should this be _local?

[heyihong]

Good catch!

[HyukjinKwon]

cc @ueshin FYI

[AmplabJenkins]

Can one of the admins verify this patch?

[HyukjinKwon]

@heyihong https://github.com/heyihong/spark/runs/5488300464?check_suite_focus=true got fixed in 54abb85. Would you mind rebasing your PR to fix the build up?

[heyihong]

@heyihong https://github.com/heyihong/spark/runs/5488300464?check_suite_focus=true got fixed in 54abb85. Would you mind rebasing your PR to fix the build up?

Sure, done

[HyukjinKwon]

@heyihong just to double check, it's just moving the codes around right? LGTM

[HyukjinKwon]

cc @ueshin mind taking a look too when you find some time?

[ueshin]

I guess we should have another file for these.

[heyihong]

Make sense, I moved the code to instrumentation_utils.py

[ueshin]

Is it possible to avoid using Anys?

[heyihong]

Make sense. Changed to stronger types

[heyihong]

@heyihong just to double check, it's just moving the codes around right? LGTM

Yes, no logic changes

[ueshin]

LGTM.

[ueshin]

Thanks! merging to master.