[Data] Unpin pyarrow from test-requirements

[scottjlee]

Why are these changes needed?

When installing ray, we found that pyarrow 6.0.1 is installed, while our default requirements.txt and other requirements files specify pyarrow >= 6.0.1 or higher. So, we try removing the pinned pyarrow==6.0.1 in test-requirements.txt to see if this is still necessary. We expect that if this is no longer needed, removing this pin will allow the resolved pyarrow version to be higher than Arrow 6, since pyarrow is already installed from the default requirements.txt, and there is a manual Arrow version override for Data tests already. This will allow users to fully utilize critical pyarrow 7+ features out of the gate, without having to manually install a higher version of Arrow.

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[krfricke]

Hi @scottjlee,

unfortunately this doesn't have an effect on the actual installed dependencies as python/requirements_compiled.txt is still pinning pyarrow.

Instead, after applying your change, you should run ./ci/ci.sh compile_pip_dependencies on a Linux machine with Python 3.8 to re-compile the dependencies.

(Ideally you can also do the same for Python 3.7 with ./ci/ci.sh compile_pip_dependencies requirements_compiled_py37.txt).

Let me know if you need any support with that.

[krfricke]

I've updated the PR with a new compiled requirements file. So far it looks good, we'll just have to see if all tests are passing.

[krfricke]

Potentially we need to re-add a pyarrow 6 test, depending on if we want to continue to test support for it.

[krfricke]

I've updated the PR and tests seem to be passing. Would be good to get another pair of eyes on cc @matthewdeng

[matthewdeng]

Looks good assuming this was generated in clean environment.