Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug: error="invalid payload: invalid json received in payload: unexpected end of JSON input" occurs after removing data in Source connector #17

Open
SerhiiSpivak opened this issue Oct 5, 2022 · 0 comments
Labels
bug Something isn't working

Comments

@SerhiiSpivak
Copy link

SerhiiSpivak commented Oct 5, 2022

Bug description

We create the Redis Destination connector and the Sorce connector (for example Stripe); then start the pipeline and remove data in Stripe -> the system returns an error after starting

2022-10-04T12:21:38+00:00 ERR error writing record error="invalid payload: invalid json received in payload: unexpected end of JSON input" component=plugin connector_id=4d38f00d-0350-4f62-b0d9-21fbd89130fd plugin_name=conduit-connector-redis record_position="{\"mode\":\"cdc\",\"created_at\":1664875257,\"cursor\":\"evt_1Lp7JlIT2rddkluqibgOvF14\",\"index\":0}"

Steps to reproduce

  1. Create Pipeline
  2. Create Stripe source connector and Redis destination connector with "mode": "stream" (see below config):
version: 1.0                  

pipelines:                    
  pipeline1:                   
    status: stopped             
    name: pipeline1             
    description: desc           
    connectors:                 
      con1:                 
        type: source            
        plugin: standalone:stripe    
        name: Stripe-source             
        settings:              
          secretKey: <Stripe Secret Key> 
          resourceName: coupon
          batchSize:
      con2:
        type: destination
        plugin: standalone:redis
        name: Redis [straem]-DESTINATION
        settings:
           redis.key: users
           redis.host: 172.17.0.3
           redis.port:
           redis.database:
           redis.username:
           redis.password:
           mode: stream
  1. Strat Pipeline
  2. Remove data in Stripe
  3. Get an error
2022-10-04T12:21:38+00:00 ERR error writing record error="invalid payload: invalid json received in payload: unexpected end of JSON input" component=plugin connector_id=4d38f00d-0350-4f62-b0d9-21fbd89130fd plugin_name=conduit-connector-redis record_position="{\"mode\":\"cdc\",\"created_at\":1664875257,\"cursor\":\"evt_1Lp7JlIT2rddkluqibgOvF14\",\"index\":0}"

additional info: same problem with other DB connectors

An error when we try to transfer data from Vitess DB to Redis

 2022-10-04T13:00:47+00:00 ERR error writing record error="invalid payload: invalid json received in payload: unexpected end of JSON input" component=plugin connector_id=3dbda980-bcc8-445b-a9aa-1fa4f45e64a2 plugin_name=conduit-connector-redis record_position="{\"mode\":\"cdc\",\"keyspace\":\"test\",\"last_processed_element_value\":402,\"shard_gtids\":[{\"keyspace\":\"test\",\"shard\":\"-40\",\"gtid\":\"MySQL56/5046c65c-4334-11ed-833d-0242ac140002:1-94\"},{\"keyspace\":\"test\",\"shard\":\"40-80\",\"gtid\":\"MySQL56/5046c65c-4334-11ed-833d-0242ac140002:1-94\"},{\"keyspace\":\"test\",\"shard\":\"80-c0\",\"gtid\":\"MySQL56/5046c65c-4334-11ed-833d-0242ac140002:1-94\"},{\"keyspace\":\"test\",\"shard\":\"c0-\",\"gtid\":\"MySQL56/5046c65c-4334-11ed-833d-0242ac140002:1-93\"}]}"

Version

v0.3.0-nightly.20220927

@SerhiiSpivak SerhiiSpivak added the bug Something isn't working label Oct 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant