ARROW-8175: [Python] Setup type checking with mypy

[xhochy]

No description provided.

[github-actions]

https://issues.apache.org/jira/browse/ARROW-8175

[xhochy]

The CI failure (C#) should be unrelated.

[alippai]

Can I help landing mypy support with anything? (I noticed that generated docs/interfaces are not in a good shape, with VSCode/PyCharm they are misleading, I assume this would help)

[xhochy]

Things that would help:

• One of @wesm, @pitrou or @jorisvandenbossche should state their opinion on this ;)
• We would need to have better support in Cython for type annotations. As the main share of our code base is in Cython and stub generation for Cython files is not really in a good shape, the majority of our API won't be typed at all.

[alippai]

So ideally a follow-up PR would generate mypy check-able code from this?

Instead of the current one:

(this is the CTRL+Q preview from PyCharm)

[xhochy]

So ideally a follow-up PR would generate mypy check-able code from this?

Instead of the current one:

(this is the CTRL+Q preview from PyCharm)

Yes but needs to be fixed on the Cython or mypy-stubs side.

[jorisvandenbossche]

What's the status of mypy support of cython? (not very familiar with this)

[alippai]

What's the status of mypy support of cython? (not very familiar with this)

Are you looking for something like this? python/mypy#8631

[jorisvandenbossche]

Maybe I was more looking for a human summary and its implications for this PR ;)

But since that is an open PR: there is right now no mypy support whatsoever for cython? So that would mean that, for now, we simply can't type the part of pyarrow that is coded in cython?
(maybe I just misinterpreted @xhochy's "Things that would help", as the listed "better support in Cython for type annotations" is not something that is actionable in pyarrow at the moment?)

[emkornfield]

@jorisvandenbossche @xhochy next steps here?

[jorisvandenbossche]

If we are OK with this being rather limited for now (limited to the part of pyarrow that is written in python, as I understand cython is not yet supported), then I am fine with moving forward with this.

[emkornfield]

I think any automation we can get is useful and we can file a follow-up JIRA to check periodically for cython. Unless I'm missing something even python only support is a strict improvement?

[jorisvandenbossche]

Yes, I think it will be a strict improvement, but it might also not be doing much validation yet, since a lot of the pyarrow objects / functions will just be typed with "Any".
Now, I am not experienced enough with mypy to make the tradeoff in this case, but so if someone wants to move this forward and update the PR, that's fine for me.

[emkornfield]

@xhochy in the interest of triaging issues, I'm going to close this for now, could you rebase/reopen when it is ready?