✨Source NASA: Migrate Python CDK to Low-code CDK

airbyte-integrations/connectors/source-nasa/Dockerfile

@@ -1,4 +1,4 @@
-FROM python:3.9.13-alpine3.15 as base
+FROM python:3.9.11-alpine3.15 as base
 
 # build and load all requirements
 FROM base as builder
@@ -34,5 +34,5 @@ COPY source_nasa ./source_nasa
 ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
 ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
 
-LABEL io.airbyte.version=0.1.1
+LABEL io.airbyte.version=0.2.0
 LABEL io.airbyte.name=airbyte/source-nasa

airbyte-integrations/connectors/source-nasa/README.md

@@ -1,35 +1,10 @@
 # Nasa Source
 
-This is the repository for the Nasa source connector, written in Python.
-For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.io/integrations/sources/nasa).
+This is the repository for the Nasa configuration based source connector.
+For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.com/integrations/sources/nasa).
 
 ## Local development
 
-### Prerequisites
-**To iterate on this connector, make sure to complete this prerequisites section.**
-
-#### Minimum Python version required `= 3.9.0`
-
-#### Build & Activate Virtual Environment and install dependencies
-From this connector directory, create a virtual environment:
-```
-python -m venv .venv
-```
-
-This will generate a virtualenv for this module in `.venv/`. Make sure this venv is active in your
-development environment of choice. To activate it from the terminal, run:
-```
-source .venv/bin/activate
-pip install -r requirements.txt
-pip install '.[tests]'
-```
-If you are in an IDE, follow your IDE's instructions to activate the virtualenv.
-
-Note that while we are installing dependencies from `requirements.txt`, you should only edit `setup.py` for your dependencies. `requirements.txt` is
-used for editable installs (`pip install -e`) to pull in Python dependencies from the monorepo and will call `setup.py`.
-If this is mumbo jumbo to you, don't worry about it, just put your deps in `setup.py` but install using `pip install -r requirements.txt` and everything
-should work as you expect.
-
 #### Building via Gradle
 You can also build the connector in Gradle. This is typically used in CI and not needed for your development workflow.
 
@@ -39,22 +14,14 @@ To build using Gradle, from the Airbyte repository root, run:
 ```
 
 #### Create credentials
-**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.io/integrations/sources/nasa)
+**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.com/integrations/sources/nasa)
 to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_nasa/spec.yaml` file.
 Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information.
 See `integration_tests/sample_config.json` for a sample config file.
 
 **If you are an Airbyte core member**, copy the credentials in Lastpass under the secret name `source nasa test creds`
 and place them into `secrets/config.json`.
 
-### Locally running the connector
-```
-python main.py spec
-python main.py check --config secrets/config.json
-python main.py discover --config secrets/config.json
-python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json
-```
-
 ### Locally running the connector docker image
 
 #### Build
@@ -79,32 +46,15 @@ docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-nasa:dev discover --co
 docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-nasa:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
 ```
 ## Testing
-Make sure to familiarize yourself with [pytest test discovery](https://docs.pytest.org/en/latest/goodpractices.html#test-discovery) to know how your test files and methods should be named.
-First install test dependencies into your virtual environment:
-```
-pip install .[tests]
-```
-### Unit Tests
-To run unit tests locally, from the connector directory run:
-```
-python -m pytest unit_tests
-```
 
-### Integration Tests
-There are two types of integration tests: Acceptance Tests (Airbyte's test suite for all source connectors) and custom integration tests (which are specific to this connector).
-#### Custom Integration tests
-Place custom tests inside `integration_tests/` folder, then, from the connector root, run
-```
-python -m pytest integration_tests
-```
 #### Acceptance Tests
-Customize `acceptance-test-config.yml` file to configure tests. See [Connector Acceptance Tests](https://docs.airbyte.io/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information.
+Customize `acceptance-test-config.yml` file to configure tests. See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information.
 If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.
-To run your integration tests with acceptance tests, from the connector root, run
+
+To run your integration tests with Docker, run:
 ```
-python -m pytest integration_tests -p integration_tests.acceptance
+./acceptance-test-docker.sh
 ```
-To run your integration tests with docker
 
 ### Using gradle to run tests
 All commands should be run from airbyte project root.

airbyte-integrations/connectors/source-nasa/acceptance-test-config.yml

@@ -1,29 +1,38 @@
 # See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference)
 # for more information about how to configure these tests
 connector_image: airbyte/source-nasa:dev
-tests:
+acceptance_tests:
   spec:
-    - spec_path: "source_nasa/spec.yaml"
+    tests:
+      - spec_path: "source_nasa/spec.yaml"
   connection:
-    - config_path: "secrets/config.json"
-      status: "succeed"
-    - config_path: "integration_tests/invalid_config.json"
-      status: "failed"
+    tests:
+      - config_path: "secrets/config.json"
+        status: "succeed"
+      - config_path: "integration_tests/invalid_config.json"
+        status: "failed"
   discovery:
-    - config_path: "secrets/config.json"
+    tests:
+      - config_path: "secrets/config.json"
   basic_read:
-    - config_path: "integration_tests/config.json"
-      configured_catalog_path: "integration_tests/configured_catalog.json"
-      empty_streams: []
-      expect_records:
-        path: "integration_tests/expected_records.jsonl"
-        extra_fields: no
-        exact_order: no
-        extra_records: yes
+    tests:
+      - config_path: "secrets/config.json"
+        configured_catalog_path: "integration_tests/configured_catalog.json"
+        empty_streams: []
+  # TODO uncomment this block to specify that the tests should assert the connector outputs the records provided in the input file a file
+  #        expect_records:
+  #          path: "integration_tests/expected_records.jsonl"
+  #          extra_fields: no
+  #          exact_order: no
+  #          extra_records: yes
   incremental:
-    - config_path: "secrets/config.json"
-      configured_catalog_path: "integration_tests/configured_catalog.json"
-      future_state_path: "integration_tests/abnormal_state.json"
+    # bypass_reason: "This connector does not implement incremental sync"
+    tests:
+      - config_path: "secrets/config.json"
+        configured_catalog_path: "integration_tests/configured_catalog.json"
+        future_state:
+          future_state_path: "integration_tests/abnormal_state.json"
   full_refresh:
-    - config_path: "secrets/config.json"
-      configured_catalog_path: "integration_tests/configured_catalog.json"
+    tests:
+      - config_path: "secrets/config.json"
+        configured_catalog_path: "integration_tests/configured_catalog.json"

airbyte-integrations/connectors/source-nasa/acceptance-test-docker.sh

@@ -1,2 +1,3 @@
 #!/usr/bin/env sh
+
 source "$(git rev-parse --show-toplevel)/airbyte-integrations/bases/connector-acceptance-test/acceptance-test-docker.sh"

airbyte-integrations/connectors/source-nasa/integration_tests/abnormal_state.json

@@ -1,5 +1,9 @@
-{
-  "nasa_apod": {
-    "date": "9999-12-31"
+[
+  {
+    "type": "STREAM",
+    "stream": {
+      "stream_state": { "date": "9999-12-31" },
+      "stream_descriptor": { "name": "nasa_apod" }
+    }
   }
-}
+]

airbyte-integrations/connectors/source-nasa/integration_tests/acceptance.py

@@ -11,4 +11,6 @@
 @pytest.fixture(scope="session", autouse=True)
 def connector_setup():
     """This fixture is a placeholder for external resources that acceptance test might require."""
+    # TODO: setup test dependencies if needed. otherwise remove the TODO comments
     yield
+    # TODO: clean up test dependencies

airbyte-integrations/connectors/source-nasa/integration_tests/expected_records.jsonl

@@ -1,6 +1 @@
-{"stream": "nasa_apod", "data": {"copyright": "Gerardo Ferrarino", "date": "2022-09-10", "explanation": "This 180 degree panoramic night skyscape captures our Milky Way Galaxy as it arcs above the horizon on a winter's night in August. Near midnight, the galactic center is close to the zenith with the clear waters of Lake Traful, Neuquen, Argentina, South America, planet Earth below. Zodiacal light, dust reflected sunlight along the Solar System's ecliptic plane, is also visible in the region's very dark night sky. The faint band of light reaches up from the distant snowy peaks toward the galaxy's center. Follow the arc of the Milky Way to the left to find the southern hemisphere stellar beacons Alpha and Beta Centauri. Close to the horizon bright star Vega is reflected in the calm mountain lake.", "hdurl": "https://apod.nasa.gov/apod/image/2209/Traful-Lake.jpg", "media_type": "image", "service_version": "v1", "title": "Galaxy by the Lake", "url": "https://apod.nasa.gov/apod/image/2209/Traful-Lake1024.jpg"}, "emitted_at": 1666637798520}
-{"stream": "nasa_apod", "data": {"date": "2022-09-11", "explanation": "How does your favorite planet spin? Does it spin rapidly around a nearly vertical axis, or horizontally, or backwards?  The featured video animates NASA images of all eight planets in our Solar System to show them spinning side-by-side for an easy comparison. In the time-lapse video, a day on Earth -- one Earth rotation -- takes just a few seconds.  Jupiter rotates the fastest, while Venus spins not only the slowest (can you see it?), but backwards.  The inner rocky planets across the top underwent dramatic spin-altering collisions during the early days of the Solar System.  Why planets spin and tilt as they do remains a topic of research with much insight gained from modern computer modeling and the recent discovery and analysis of hundreds of exoplanets: planets orbiting other stars.", "media_type": "video", "service_version": "v1", "title": "Planets of the Solar System: Tilts and Spins", "url": "https://www.youtube.com/embed/my1euFQHH-o?rel=0"}, "emitted_at": 1666637798523}
-{"stream": "nasa_apod", "data": {"copyright": "Daniel \u0160\u010derba", "date": "2022-09-12", "explanation": "What are those red filaments in the sky? They are a rarely seen form of lightning confirmed only about 35 years ago: red sprites. Research has shown that following a powerful positive cloud-to-ground lightning strike, red sprites may start as 100-meter balls of ionized air that shoot down from about 80-km high at 10 percent the speed of light. They are quickly followed by a group of upward streaking ionized balls. The featured image was taken late last month from the Jeseniky Mountains in northern Moravia in the Czech Republic. The distance to the red sprites is about 200 kilometers. Red sprites take only a fraction of a second to occur and are best seen when powerful thunderstorms are visible from the side.    APOD in world languages: Arabic, Bulgarian, Catalan, Chinese (Beijing), Chinese (Taiwan), Croatian, Czech, Dutch, Farsi, French, French (Canada), German, Hebrew, Indonesian, Japanese, Korean, Montenegrin, Polish, Russian, Serbian, Slovenian,  Spanish, Taiwanese, Turkish, and  Ukrainian", "hdurl": "https://apod.nasa.gov/apod/image/2209/sprites_scerba_4240.jpg", "media_type": "image", "service_version": "v1", "title": "Red Sprite Lightning over the Czech Republic", "url": "https://apod.nasa.gov/apod/image/2209/sprites_scerba_960.jpg"}, "emitted_at": 1666637798524}
-{"stream": "nasa_apod", "data": {"copyright": "Alan FriedmanAverted Imagination", "date": "2022-09-13", "explanation": "rlier this month, the Sun exhibited one of the longer filaments on record.  Visible as the bright curving streak around the image center, the snaking filament's full extent was estimated to be over half of the Sun's radius -- more than 350,000 kilometers long. A filament is composed of  hot gas held aloft by the Sun's magnetic field, so that viewed from the side it would appear as a raised prominence.  A different, smaller prominence is simultaneously visible at the Sun's edge. The featured image is in false-color and color-inverted to highlight not only the filament but the Sun's carpet chromosphere.  The bright dot on the upper right is actually a dark sunspot about the size of the Earth.  Solar filaments typically last from hours to days, eventually collapsing to return hot plasma back to the Sun. Sometimes, though, they explode and expel particles into the Solar System, some of which trigger auroras on Earth. The pictured filament appeared in early September and continued to hold steady for about a week.", "hdurl": "https://apod.nasa.gov/apod/image/2209/SnakingFilament_Friedman_960.jpg", "media_type": "image", "service_version": "v1", "title": "A Long Snaking Filament on the Sun", "url": "https://apod.nasa.gov/apod/image/2209/SnakingFilament_Friedman_960.jpg"}, "emitted_at": 1666637798524}
-{"stream": "nasa_apod", "data": {"copyright": "Jarmo Ruuth Text: Ata SarajediniFlorida Atlantic U.Astronomy Minute", "date": "2022-09-14", "explanation": "It is one of the largest nebulas on the sky -- why isn't it better known? Roughly the same angular size as the Andromeda Galaxy, the Great Lacerta Nebula can be found toward the constellation of the Lizard (Lacerta). The emission nebula is difficult to see with wide-field binoculars because it is so faint, but also usually difficult to see with a large telescope because it is so great in angle -- spanning about three degrees. The depth, breadth, waves, and beauty of the nebula -- cataloged as Sharpless 126 (Sh2-126) -- can best be seen and appreciated with a long duration camera exposure. The featured image is one such combined exposure -- in this case 10 hours over five different colors and over six nights during this past June and July at the IC Astronomy Observatory in Spain. The hydrogen gas in the Great Lacerta Nebula glows red because it is excited by light from the bright star 10 Lacertae, one of the bright blue stars just above the red-glowing nebula's center.  The stars and nebula are about 1,200 light years distant.    Harvest Full Moon 2022: Notable Submissions to APOD", "hdurl": "https://apod.nasa.gov/apod/image/2209/GreatLacerta_Ruuth_3719.jpg", "media_type": "image", "service_version": "v1", "title": "Waves of the Great Lacerta Nebula", "url": "https://apod.nasa.gov/apod/image/2209/GreatLacerta_Ruuth_960.jpg"}, "emitted_at": 1666637798524}
-{"stream": "nasa_apod", "data": {"copyright": "Dario Giannobile", "date": "2022-09-15", "explanation": "For northern hemisphere dwellers, September's Full Moon was the Harvest Moon. Reflecting warm hues at sunset it rises over the historic town of Castiglione di Sicilia in this telephoto view from September 9. Famed in festival, story, and song Harvest Moon is just the traditional name of the full moon nearest the autumnal equinox. According to lore the name is a fitting one. Despite the diminishing daylight hours as the growing season drew to a close, farmers could harvest crops by the light of a full moon shining on from dusk to dawn.   Harvest Full Moon 2022: Notable Submissions to APOD", "hdurl": "https://apod.nasa.gov/apod/image/2209/HarvestMoonCastiglioneSicilyLD.jpg", "media_type": "image", "service_version": "v1", "title": "Harvest Moon over Sicily", "url": "https://apod.nasa.gov/apod/image/2209/HarvestMoonCastiglioneSicily1024.jpg"}, "emitted_at": 1666637798525}
+{"stream": "nasa_apod", "data": {"copyright":"\nAbdullahAlharbi\n","date":"2023-10-04","explanation":"Doesthisnebulalookliketheheadofawitch?ThenebulaisknownpopularlyastheWitchHeadNebulabecause,itissaid,thenebula'sshaperesemblesaHalloween-stylecaricatureofawitch'shead.Exactlyhow,though,canbeatopicofimaginativespeculation.WhatisclearisthatIC2118isabout50light-yearsacrossandmadeofgasanddustthatpointsto--becauseithasbeenpartlyerodedby--thenearbystarRigel.OneofthebrighterstarsintheconstellationOrion,Rigelliesbelowthebottomofthefeaturedimage.ThebluecoloroftheWitchHeadNebulaandiscausednotonlybyRigel'sintensebluestarlightbutbecausethedustgrainsscatterbluelightmoreefficientlythanred.ThesamephysicalprocesscausesEarth'sdaytimeskytoappearblue,althoughthescatterersinplanetEarth'satmospherearemoleculesofnitrogenandoxygen.","hdurl":"https://apod.nasa.gov/apod/image/2310/WitchHead_Alharbi_3051.jpg","media_type":"image","service_version":"v1","title":"IC2118:TheWitchHeadNebula","url":"https://apod.nasa.gov/apod/image/2310/WitchHead_Alharbi_1080.jpg"}, "emitted_at": 1666637798524}

airbyte-integrations/connectors/source-nasa/integration_tests/sample_config.json

@@ -1,5 +1,5 @@
 {
-  "api_key": "DEMO_KEY",
-  "date": "2022-10-20",
-  "concept_tags": true
+  "api_key": "xxxxxxxxxxxxxxxxxxxxxxxx",
+  "start_date": "2023-10-01",
+  "end_date": "2023-10-03"
 }

airbyte-integrations/connectors/source-nasa/integration_tests/sample_state.json

@@ -1,5 +1,13 @@
-{
-  "nasa-apod": {
-    "date": "2022-10-15"
+[
+  {
+    "type": "STREAM",
+    "stream": {
+      "stream_descriptor": {
+        "name": "nasa_apod"
+      },
+      "stream_state": {
+        "date": "2023-10-20"
+      }
+    }
   }
-}
+]

airbyte-integrations/connectors/source-nasa/metadata.yaml

@@ -2,26 +2,24 @@ data:
   allowedHosts:
     hosts:
       - api.nasa.gov
+  registries:
+    oss:
+      enabled: true
+    cloud:
+      enabled: false
   connectorSubtype: api
   connectorType: source
   definitionId: 1a8667d7-7978-43cd-ba4d-d32cbd478971
-  dockerImageTag: 0.1.1
+  dockerImageTag: 0.2.0
   dockerRepository: airbyte/source-nasa
   githubIssueLabel: source-nasa
   icon: nasa.svg
   license: MIT
-  name: NASA
-  registries:
-    cloud:
-      enabled: false
-    oss:
-      enabled: true
+  name: Nasa
+  releaseDate: 2023-10-09
   releaseStage: alpha
+  supportLevel: community
   documentationUrl: https://docs.airbyte.com/integrations/sources/nasa
   tags:
-    - language:python
-  ab_internal:
-    sl: 100
-    ql: 100
-  supportLevel: community
+    - language:lowcode
 metadataSpecVersion: "1.0"

airbyte-integrations/connectors/source-nasa/requirements.txt

@@ -1 +1,2 @@
+-e ../../bases/connector-acceptance-test
 -e .

airbyte-integrations/connectors/source-nasa/setup.py

@@ -5,15 +5,9 @@
 
 from setuptools import find_packages, setup
 
-MAIN_REQUIREMENTS = [
-    "airbyte-cdk~=0.2",
-]
+MAIN_REQUIREMENTS = ["airbyte-cdk"]
 
-TEST_REQUIREMENTS = [
-    "requests-mock~=1.9.3",
-    "pytest~=6.1",
-    "pytest-mock~=3.6.1",
-]
+TEST_REQUIREMENTS = ["requests-mock~=1.9.3", "pytest~=6.2", "pytest-mock~=3.6.1"]
 
 setup(
     name="source_nasa",