Python: Add adlfs support (Azure DataLake FileSystem)

[cccs-eric]

Adds support for adlfs storage in Python API.

[rdblue]

Looks good to me overall. I just have one comment about the FileNotFoundError. I'll defer to @Fokko to also review since he catches a lot more than I do in Python.

[Fokko]

One minor comment about the credential, but apart from that this is looks great 👏🏻

[Fokko]

@cccs-eric it looks like we have to add pyparsing to the pyproject.toml. I don't know why it was working before, but it should have been added in #6259

[Fokko]

@cccs-eric I forgot one thing, could you also add the adlfs option to the docs in python/mkdocs/?

[cccs-eric]

Yes, I'll do that. And I've also realized that I'm not calling the tests for ADLFS. Should I simply add this to python-ci.yml?

    - name: Tests S3
      working-directory: ./python
      run: make test-s3
    - name: Tests Adlfs
      working-directory: ./python
      run: make test-adlfs

[Fokko]

@cccs-eric Yes, please do!

[cccs-eric]

@cccs-eric it looks like we have to add pyparsing to the pyproject.toml. I don't know why it was working before, but it should have been added in #6259

@Fokko I don't understand what is going on. pyparsing is part of pyproject.toml and when I run make test-s3, it works on my machine ™️

[rdblue]

@cccs-eric, I just merged $6439 that fixes the pyparsing problem (my fault) so you should be able to rebase and get tests working. Sorry about that!

[cccs-eric]

@cccs-eric, I just merged $6439 that fixes the pyparsing problem (my fault) so you should be able to rebase and get tests working. Sorry about that!

Thanks @rdblue , build is now passing 🥳

[cccs-eric]

@Fokko One last thing that I haven't done is modifying verify-release.md for the integration tests. Should I add test-adlfs in there?
https://github.com/apache/iceberg/blob/master/python/mkdocs/docs/verify-release.md?plain=1#L86-L90

[rdblue]

@cccs-eric, looks like this was out of date. I tried to fix conflicts, but it is failing tests. Can you fix and then we'll merge?

[cccs-eric]

@rdblue All clear now

[Fokko]

Awesome, thanks @cccs-eric for working on this 🚀

[rdblue]

Thanks, @cccs-eric!