Test file formats #1392 #1768

vitaliizazmic · 2021-01-21T20:23:02Z

How

Create file samples
Check uploading each of them from Local storage
Remove html, orc and pickle from available formats in spec.json
Create share test case for all formats to pull file

Pre-merge Checklist

Run integration tests
Publish Docker images

vitaliizazmic · 2021-01-21T20:24:37Z

/test connector=source-file

🕑 source-file https://github.com/airbytehq/airbyte/actions/runs/502087916
✅ source-file https://github.com/airbytehq/airbyte/actions/runs/502087916
🕑 source-file https://github.com/airbytehq/airbyte/actions/runs/502087916
✅ source-file https://github.com/airbytehq/airbyte/actions/runs/502087916

…le format

…supported file formats

sherifnada · 2021-01-22T23:44:27Z

@vitaliizazmic I added custom integration tests to CI -- can you make sure those tests are succeeding? you can also run them locally by running python -m pytest -s integration_tests from the module directory with the venv activated, will be much faster for iteration

…nd aws providers test

…' into vitalii/1392_test_file_formats

sherifnada · 2021-01-27T03:36:49Z

@vitaliizazmic could you pull in @eugene-kulak 's changes from #1712 and ping me when this is ready for review? All the changes he needs to make are stylistic so I think you should be able to build on top of it. One thing to keep in mind is that you should probably put your integration tests in a separate test file for easier reading.

# Conflicts: # airbyte-integrations/connectors/source-file/build.gradle # airbyte-integrations/connectors/source-file/integration_tests/integration_source_test.py # airbyte-integrations/connectors/source-file/setup.py # docs/integrations/sources/file.md

sherifnada · 2021-01-29T06:53:37Z

/test connector=source-file

🕑 source-file https://github.com/airbytehq/airbyte/actions/runs/520191436
✅ source-file https://github.com/airbytehq/airbyte/actions/runs/520191436
🕑 source-file https://github.com/airbytehq/airbyte/actions/runs/520191436
✅ source-file https://github.com/airbytehq/airbyte/actions/runs/520191436

sherifnada

small comments, will merge tomorrow

sherifnada · 2021-01-29T06:54:45Z

docs/integrations/sources/file.md

@@ -67,7 +67,7 @@ File Formats are mostly enabled \(and further tested\) thanks to other open-sour
 | :--- | :--- |
 | CSV | Yes |
 | JSON | Experimental |


why is this one still listed as experimental?

sherifnada · 2021-01-29T06:55:26Z

...ors/source-file/integration_tests/sample_files/formats/pickle/configured_catalog_pickle.json

@@ -0,0 +1,21 @@
+{


if we don't support parquet/orc/pickle can we remove them and any catalogs/sample files that are not supported?

@sherifnada we support parquet and pickle

Pickle works if it contains DataFrame. Because read_pickle() method returns same type as object stored in file. And discover and read methods of client expect for DataFrame.

sherifnada · 2021-01-29T06:57:45Z

airbyte-integrations/connectors/source-file/setup.py

@@ -34,6 +34,12 @@
    "paramiko==2.7.2",
    "s3fs==0.5.2",
    "smart-open[all]==4.1.2",
+    "lxml==4.6.2",
+    "html5lib==1.1",
+    "beautifulsoup4==4.9.3",


are all of these libs required? html/BS/pyarrow?

Yes. This libraries required for read different file formats.

sherifnada · 2021-01-29T07:04:11Z

@vitaliizazmic once you've addressed the comments could you:

pull master ( i merged eugene's bugfix for source-file)
bump the connector version
run the test command

I'll publish and merge. Thanks!

ChristopheDuong · 2021-01-29T15:33:51Z

docs/integrations/sources/file.md

@@ -66,13 +66,13 @@ File Formats are mostly enabled \(and further tested\) thanks to other open-sour
 | Format | Supported? |
 | :--- | :--- |
 | CSV | Yes |
-| JSON | Experimental |
-| HTML | Untested |
+| JSON | Yes |


I originally put it as 'Experimental' because Source-File couldn't handle nested JSON files
(flat JSON file worked)

sherifnada · 2021-02-01T22:47:36Z

/publish connector=connectors=source-file

Error: Workflow does not have 'workflow_dispatch' trigger

…92_test_file_formats

sherifnada · 2021-02-01T22:49:54Z

/publish connector=connectors=source-file

❌ connectors=source-file https://github.com/airbytehq/airbyte/actions/runs/529124974

sherifnada · 2021-02-01T22:59:17Z

/publish connector=connectors/source-file

🕑 connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/529143991
❌ connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/529143991

sherifnada · 2021-02-01T23:14:35Z

/test connector=source-file

🕑 source-file https://github.com/airbytehq/airbyte/actions/runs/529174443
❌ source-file https://github.com/airbytehq/airbyte/actions/runs/529174443
🕑 source-file https://github.com/airbytehq/airbyte/actions/runs/529174443
❌ source-file https://github.com/airbytehq/airbyte/actions/runs/529174443
🕑 source-file https://github.com/airbytehq/airbyte/actions/runs/529174443
❌ source-file https://github.com/airbytehq/airbyte/actions/runs/529174443

sherifnada · 2021-02-01T23:15:53Z

@vitaliizazmic tests are failing, can you take a look ?

…' into vitalii/1392_test_file_formats

vitaliizazmic · 2021-02-02T21:52:58Z

/test connector=source-file

🕑 source-file https://github.com/airbytehq/airbyte/actions/runs/532068832
✅ source-file https://github.com/airbytehq/airbyte/actions/runs/532068832

sherifnada · 2021-02-02T22:37:03Z

/publish connector=connectors/source-file

🕑 connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/532163094
✅ connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/532163094

Test file formats #1392 - creating samples and test case

79d2d42

vitaliizazmic and others added 4 commits January 22, 2021 00:58

Test file formats #1392 - removing html form from supported

2929a7e

Google ads best practices #1568 - configured_catalog.json for each fi…

84ebf19

…le format

Test file formats #1392 - disabling integration test for temporary un…

ec925d9

…supported file formats

add custom integration tests to CI

b4bfaa0

vitaliizazmic added 2 commits January 25, 2021 07:45

Test file formats #1392 - fixing configs file path for existing gcs a…

0499905

…nd aws providers test

Merge remote-tracking branch 'upstream/vitalii/1392_test_file_formats…

fb454da

…' into vitalii/1392_test_file_formats

vitaliizazmic added 3 commits January 28, 2021 00:18

Merge branch 'master' into vitalii/1392_test_file_formats

099bd6b

File source #1392 - merging with best practice

c14d78a

vitaliizazmic requested a review from sherifnada January 28, 2021 16:48

sherifnada approved these changes Jan 29, 2021

View reviewed changes

vitaliizazmic added 2 commits January 29, 2021 16:18

Merge branch 'master' into vitalii/1392_test_file_formats

6e964a0

File source #1392 - removing samples, bumping version

bf60fc1

ChristopheDuong reviewed Jan 29, 2021

View reviewed changes

Merge branch 'master' of github.com:airbytehq/airbyte into vitalii/13…

8338b3c

…92_test_file_formats

vitaliizazmic added 3 commits February 2, 2021 23:21

File source #1392 - updating pyarrow version

f7d536d

Merge remote-tracking branch 'upstream/vitalii/1392_test_file_formats…

dd87fe7

…' into vitalii/1392_test_file_formats

File source #1392 - decrease pandas version to 1.2.0

ffdda03

sherifnada merged commit caf673d into master Feb 2, 2021

sherifnada deleted the vitalii/1392_test_file_formats branch February 2, 2021 22:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test file formats #1392 #1768

Test file formats #1392 #1768

vitaliizazmic commented Jan 21, 2021 •

edited

Loading

vitaliizazmic commented Jan 21, 2021 •

edited by github-actions bot

Loading

sherifnada commented Jan 22, 2021

sherifnada commented Jan 27, 2021 •

edited

Loading

sherifnada commented Jan 29, 2021 •

edited by github-actions bot

Loading

sherifnada left a comment

sherifnada Jan 29, 2021

sherifnada Jan 29, 2021

eugene-kulak Jan 29, 2021

vitaliizazmic Jan 29, 2021

sherifnada Jan 29, 2021

vitaliizazmic Jan 29, 2021

sherifnada commented Jan 29, 2021

ChristopheDuong Jan 29, 2021

sherifnada commented Feb 1, 2021 •

edited by github-actions bot

Loading

sherifnada commented Feb 1, 2021 •

edited by github-actions bot

Loading

sherifnada commented Feb 1, 2021 •

edited by github-actions bot

Loading

sherifnada commented Feb 1, 2021 •

edited by github-actions bot

Loading

sherifnada commented Feb 1, 2021

vitaliizazmic commented Feb 2, 2021 •

edited by github-actions bot

Loading

sherifnada commented Feb 2, 2021 •

edited by github-actions bot

Loading

Test file formats #1392 #1768

Test file formats #1392 #1768

Conversation

vitaliizazmic commented Jan 21, 2021 • edited Loading

How

Pre-merge Checklist

vitaliizazmic commented Jan 21, 2021 • edited by github-actions bot Loading

sherifnada commented Jan 22, 2021

sherifnada commented Jan 27, 2021 • edited Loading

sherifnada commented Jan 29, 2021 • edited by github-actions bot Loading

sherifnada left a comment

Choose a reason for hiding this comment

sherifnada Jan 29, 2021

Choose a reason for hiding this comment

sherifnada Jan 29, 2021

Choose a reason for hiding this comment

eugene-kulak Jan 29, 2021

Choose a reason for hiding this comment

vitaliizazmic Jan 29, 2021

Choose a reason for hiding this comment

sherifnada Jan 29, 2021

Choose a reason for hiding this comment

vitaliizazmic Jan 29, 2021

Choose a reason for hiding this comment

sherifnada commented Jan 29, 2021

ChristopheDuong Jan 29, 2021

Choose a reason for hiding this comment

sherifnada commented Feb 1, 2021 • edited by github-actions bot Loading

sherifnada commented Feb 1, 2021 • edited by github-actions bot Loading

sherifnada commented Feb 1, 2021 • edited by github-actions bot Loading

sherifnada commented Feb 1, 2021 • edited by github-actions bot Loading

sherifnada commented Feb 1, 2021

vitaliizazmic commented Feb 2, 2021 • edited by github-actions bot Loading

sherifnada commented Feb 2, 2021 • edited by github-actions bot Loading

vitaliizazmic commented Jan 21, 2021 •

edited

Loading

vitaliizazmic commented Jan 21, 2021 •

edited by github-actions bot

Loading

sherifnada commented Jan 27, 2021 •

edited

Loading

sherifnada commented Jan 29, 2021 •

edited by github-actions bot

Loading

sherifnada commented Feb 1, 2021 •

edited by github-actions bot

Loading

sherifnada commented Feb 1, 2021 •

edited by github-actions bot

Loading

sherifnada commented Feb 1, 2021 •

edited by github-actions bot

Loading

sherifnada commented Feb 1, 2021 •

edited by github-actions bot

Loading

vitaliizazmic commented Feb 2, 2021 •

edited by github-actions bot

Loading

sherifnada commented Feb 2, 2021 •

edited by github-actions bot

Loading