🎉 Source BambooHR: Add custom reports endpoint

[olivermeyer]

What

In this PR I add support for the custom reports endpoint to the BambooHR source connector. In order to do so, I refactor the existing code to fit the current implementation of the CDK.

The custom reports endpoint behaves slightly differently than most endpoints do, in that it expects a list of fields and returns only those fields. I expose that list of fields through two configuration options:

• custom_reports_fields is a list of fields to be included in the request
• custom_reports_include_default_fields determines whether default fields are included in the request, in addition to the fields specified in custom_reports_fields

🚨 User Impact 🚨

The existing employee stream has been renamed to employees_directory_stream.

Pre-merge Checklist

Community member or Airbyter

• Grant edit access to maintainers (instructions)
• Secrets in the connector's spec are annotated with airbyte_secret
• Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• Code reviews completed
• Documentation updated
  • Connector's README.md
  • Connector's bootstrap.md. See description and examples
  • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
• PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• If new credentials are required for use in CI, add them to GSM. Instructions.
• /test connector=connectors/<name> command is passing
• New Connector version released on Dockerhub by running the /publish command described here
• After the new connector version is published, connector version bumped in the seed directory as described here
• Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

[olivermeyer]

Output of unit tests:

$ python -m pytest unit_tests
Test session starts (platform: darwin, Python 3.7.9, pytest 6.2.5, pytest-sugar 0.9.4)
cachedir: .pytest_cache
rootdir: /Users/Omeyer/code/perso/airbyte, configfile: pytest.ini
plugins: sugar-0.9.4, timeout-1.4.2
collecting ...
 airbyte-integrations/connectors/source-bamboo-hr/unit_tests/unit_test.py::test_source_bamboo_hr_client_wrong_credentials ✓                                                                   20% ██
 airbyte-integrations/connectors/source-bamboo-hr/unit_tests/unit_test.py::test_employees_directory_stream_url_base ✓                                                                         40% ████
 airbyte-integrations/connectors/source-bamboo-hr/unit_tests/unit_test.py::test_custom_reports_stream_get_json_schema_from_config ✓                                                           60% ██████
 airbyte-integrations/connectors/source-bamboo-hr/unit_tests/unit_test.py::test_custom_reports_stream_union_schemas ✓                                                                         80% ████████
 airbyte-integrations/connectors/source-bamboo-hr/unit_tests/unit_test.py::test_custom_reports_stream_request_body_json ✓                                                                    100% ██████████
============================================================================================= warnings summary =============================================================================================
airbyte-integrations/connectors/source-bamboo-hr/unit_tests/unit_test.py::test_employees_directory_stream_url_base
airbyte-integrations/connectors/source-bamboo-hr/unit_tests/unit_test.py::test_custom_reports_stream_get_json_schema_from_config
airbyte-integrations/connectors/source-bamboo-hr/unit_tests/unit_test.py::test_custom_reports_stream_request_body_json
  /Users/Omeyer/code/perso/airbyte/airbyte-integrations/connectors/source-bamboo-hr/.venv/lib/python3.7/site-packages/airbyte_cdk/sources/streams/http/http.py:42: DeprecationWarning: Call to deprecated class NoAuth. (Set `authenticator=None` instead) -- Deprecated since version 0.1.20.
    self._authenticator: HttpAuthenticator = NoAuth()

airbyte-integrations/connectors/source-bamboo-hr/unit_tests/unit_test.py::test_employees_directory_stream_url_base
airbyte-integrations/connectors/source-bamboo-hr/unit_tests/unit_test.py::test_custom_reports_stream_get_json_schema_from_config
airbyte-integrations/connectors/source-bamboo-hr/unit_tests/unit_test.py::test_custom_reports_stream_request_body_json
  /Users/Omeyer/code/perso/airbyte/airbyte-integrations/connectors/source-bamboo-hr/.venv/lib/python3.7/site-packages/deprecated/classic.py:173: DeprecationWarning: Call to deprecated class HttpAuthenticator. (Use requests.auth.AuthBase instead) -- Deprecated since version 0.1.20.
    return old_new1(cls, *args, **kwargs)

-- Docs: https://docs.pytest.org/en/stable/warnings.html

Results (0.67s):
       5 passed

Output of integration tests:

$ python -m pytest integration_tests

Test session starts (platform: darwin, Python 3.7.9, pytest 6.2.5, pytest-sugar 0.9.4)
cachedir: .pytest_cache
rootdir: /Users/Omeyer/code/perso/airbyte, configfile: pytest.ini
plugins: sugar-0.9.4, timeout-1.4.2
collecting ...

Results (0.02s):

I'm not sure why there are no integration tests.

[marcosmarxm]

@olivermeyer please share the output of ./gradlew :airbyte-integrations:connectors:source-bamboo-hr:integrationTest

[olivermeyer]

@marcosmarxm please see below. It's failing for :airbyte-commons:compileJava. I've never worked with gradle so I have no idea how to troubleshoot this.

$ ./gradlew :airbyte-integrations:connectors:source-bamboo-hr:integrationTest
Building all of Airbyte.
/Users/Omeyer/code/perso/airbyte/airbyte-integrations/connectors
> Task :airbyte-commons:compileJava FAILED

> Task :airbyte-integrations:bases:base-normalization:checkPython
Using python 3.7.9 from /Users/Omeyer/code/perso/airbyte/airbyte-integrations/bases/base-normalization/.venv (.venv/bin/python)
Using pip 21.3.1 from /Users/Omeyer/code/perso/airbyte/airbyte-integrations/bases/base-normalization/.venv/lib/python3.7/site-packages/pip (python 3.7)

> Task :airbyte-integrations:bases:base-python:checkPython
Using python 3.7.9 from /Users/Omeyer/code/perso/airbyte/airbyte-integrations/bases/base-python/.venv (.venv/bin/python)
Using pip 21.3.1 from /Users/Omeyer/code/perso/airbyte/airbyte-integrations/bases/base-python/.venv/lib/python3.7/site-packages/pip (python 3.7)

> Task :airbyte-integrations:bases:base-singer:checkPython
Using python 3.7.9 from /Users/Omeyer/code/perso/airbyte/airbyte-integrations/bases/base-singer/.venv (.venv/bin/python)
Using pip 21.3.1 from /Users/Omeyer/code/perso/airbyte/airbyte-integrations/bases/base-singer/.venv/lib/python3.7/site-packages/pip (python 3.7)

> Task :airbyte-integrations:bases:source-acceptance-test:checkPython
Using python 3.7.9 from /Users/Omeyer/code/perso/airbyte/airbyte-integrations/bases/source-acceptance-test/.venv (.venv/bin/python)
Using pip 21.3.1 from /Users/Omeyer/code/perso/airbyte/airbyte-integrations/bases/source-acceptance-test/.venv/lib/python3.7/site-packages/pip (python 3.7)

> Task :airbyte-integrations:connectors:source-bamboo-hr:checkPython
Using python 3.7.9 from /Users/Omeyer/code/perso/airbyte/airbyte-integrations/connectors/source-bamboo-hr/.venv (.venv/bin/python)
Using pip 21.3.1 from /Users/Omeyer/code/perso/airbyte/airbyte-integrations/connectors/source-bamboo-hr/.venv/lib/python3.7/site-packages/pip (python 3.7)

> Task :airbyte-integrations:bases:airbyte-protocol:checkPython
Using python 3.7.9 from /Users/Omeyer/code/perso/airbyte/airbyte-integrations/bases/airbyte-protocol/.venv (.venv/bin/python)
Using pip 21.3.1 from /Users/Omeyer/code/perso/airbyte/airbyte-integrations/bases/airbyte-protocol/.venv/lib/python3.7/site-packages/pip (python 3.7)

> Task :airbyte-integrations:bases:base-python-test:checkPython
Using python 3.7.9 from /Users/Omeyer/code/perso/airbyte/airbyte-integrations/bases/base-python-test/.venv (.venv/bin/python)
Using pip 21.3.1 from /Users/Omeyer/code/perso/airbyte/airbyte-integrations/bases/base-python-test/.venv/lib/python3.7/site-packages/pip (python 3.7)

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':airbyte-commons:compileJava'.
> error: invalid source release: 17

[marcosmarxm]

@olivermeyer can you install Java 17?

[olivermeyer]

@olivermeyer can you install Java 17?

@marcosmarxm that helped, thanks. Below the output of the unit tests after the latest changes:

$ python -m pytest unit_tests
Test session starts (platform: darwin, Python 3.7.9, pytest 6.1.2, pytest-sugar 0.9.4)
cachedir: .pytest_cache
rootdir: /Users/Omeyer/code/perso/airbyte, configfile: pytest.ini
plugins: sugar-0.9.4, timeout-1.4.2
collecting ...
 airbyte-integrations/connectors/source-bamboo-hr/unit_tests/unit_test.py::test_source_bamboo_hr_client_wrong_credentials ✓                                                                                                                                                                                                                                 20% ██
 airbyte-integrations/connectors/source-bamboo-hr/unit_tests/unit_test.py::test_employees_directory_stream_url_base ✓                                                                                                                                                                                                                                       40% ████
 airbyte-integrations/connectors/source-bamboo-hr/unit_tests/unit_test.py::test_custom_reports_stream_get_json_schema_from_config ✓                                                                                                                                                                                                                         60% ██████
 airbyte-integrations/connectors/source-bamboo-hr/unit_tests/unit_test.py::test_custom_reports_stream_union_schemas ✓                                                                                                                                                                                                                                       80% ████████
 airbyte-integrations/connectors/source-bamboo-hr/unit_tests/unit_test.py::test_custom_reports_stream_request_body_json ✓                                                                                                                                                                                                                                  100% ██████████
============================================================================================================================================================================ warnings summary ============================================================================================================================================================================
airbyte-integrations/connectors/source-bamboo-hr/unit_tests/unit_test.py::test_employees_directory_stream_url_base
airbyte-integrations/connectors/source-bamboo-hr/unit_tests/unit_test.py::test_custom_reports_stream_get_json_schema_from_config
airbyte-integrations/connectors/source-bamboo-hr/unit_tests/unit_test.py::test_custom_reports_stream_request_body_json
  /Users/Omeyer/code/perso/airbyte/airbyte-integrations/connectors/source-bamboo-hr/.venv/lib/python3.7/site-packages/airbyte_cdk/sources/streams/http/http.py:42: DeprecationWarning: Call to deprecated class NoAuth. (Set `authenticator=None` instead) -- Deprecated since version 0.1.20.
    self._authenticator: HttpAuthenticator = NoAuth()

airbyte-integrations/connectors/source-bamboo-hr/unit_tests/unit_test.py::test_employees_directory_stream_url_base
airbyte-integrations/connectors/source-bamboo-hr/unit_tests/unit_test.py::test_custom_reports_stream_get_json_schema_from_config
airbyte-integrations/connectors/source-bamboo-hr/unit_tests/unit_test.py::test_custom_reports_stream_request_body_json
  /Users/Omeyer/code/perso/airbyte/airbyte-integrations/connectors/source-bamboo-hr/.venv/lib/python3.7/site-packages/deprecated/classic.py:173: DeprecationWarning: Call to deprecated class HttpAuthenticator. (Use requests.auth.AuthBase instead) -- Deprecated since version 0.1.20.
    return old_new1(cls, *args, **kwargs)

-- Docs: https://docs.pytest.org/en/stable/warnings.html

Results (0.39s):
       5 passed

And the integration tests (see file below for full output):

$ ./gradlew :airbyte-integrations:connectors:source-bamboo-hr:integrationTest
Building all of Airbyte.
...
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings

Execution optimizations have been disabled for 1 invalid unit(s) of work during this build to ensure correctness.
Please consult deprecation warnings for more details.

BUILD SUCCESSFUL in 1m 22s
83 actionable tasks: 46 executed, 37 up-to-date

source-bamboo-hr-integration-tests.txt