Skip to content

This data pipeline demo repo has sample codes and commands to automatically process data files written to COS buckets and push data records to event stream with serverless code engine jobs. It also has sample consumer to consume the data records from event stream and further process the data before saving them back to COS bucket.

License

Notifications You must be signed in to change notification settings

IBM/demo-data-pipeline-w-code-engine

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

5 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

demo-data-pipeline-w-code-engine

This demo shows how data files written to COS buckets can be automatically processed through the data pipeline using services in IBM Cloud. The following steps are involved in the data pipeline:

  • Data file is uploaded to COS bucket.
  • Code engine job is triggered by COS write event (IBM Cloud container registry holds the image for the job).
  • Code engine job reads data file from COS bucket, and push data records to Kafka topic (producer). Note that this step is optional in batch data pipeline, and code engine jobs can process the data and make it ready for consumption. It is included here to demonstrate how to push data records to Kafka, since some 3rd party software/services has connector to Kafka already.
  • Data handler (consumer) consumes data records from Kafka topic, process the data if needed, and save the result back to COS bucket
  • Various applications can make use of the data

Refer to this architecture diagram.

For more detailed description, please refer to the blog post.

For quick command reference, please refer to the sample commands in this project.

About

This data pipeline demo repo has sample codes and commands to automatically process data files written to COS buckets and push data records to event stream with serverless code engine jobs. It also has sample consumer to consume the data records from event stream and further process the data before saving them back to COS bucket.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published