Exporting/importing AVRO Schemas files

[keyankay]

How can I export/import AVRO schemas from/to the schema-registry to/from an AVSC file?

[OneCricketeer]

Install jq

Export

$ export TOPIC=topicName
$ curl http://kafka-schema:8081/subjects/$TOPIC-value/versions/latest | jq -r '.schema|fromjson' > $TOPIC-value.avsc

Repeat for key schema, if needed

Import

Get the schema string

$ export SCHEMA=$(jq tostring $TOPIC-value.avsc)

echo to verify output

$ echo "{\"schema\":$SCHEMA}" 
{"schema":"{\"type\":\"record\",\"name\":...}

Post the schema

$ curl -XPOST -H "Content-Type:application/json" -d"{\"schema\":$SCHEMA}" http://kafka-schema:8081/subjects/$TOPIC-value/versions

[lanceon]

Thanks for pointing to jq, works this way.

[niqdev]

Alternative solution with HTTPie and jq

# import schema from file
http -v POST :8081/subjects/example.with-schema.user-value/versions \
  Accept:application/vnd.schemaregistry.v1+json \
  schema=@avro/src/main/avro/user.avsc

# export schema to file
http :8081/subjects/example.with-schema.user-value/versions/latest \
  | jq -r '.schema|fromjson' \
  | tee avro/src/main/avro/user-latest.avsc

You can check the full example here

[OneCricketeer]

In that repo, I would suggest trying to download as part of the SBT compile phase,much like how you can use the Maven plugin otherwise

[niqdev]

I'm looking for sbt alternatives to kafka-schema-registry-maven-plugin. @Cricket007 anything to recommend?

So far, I found sbt-schema-registry-downloader, but is not production ready and if I don't find one, I was thinking to write a plugin that does the same as the maven plugin (probably using http4s instead of Akka) that can be integrated with sbt-avrohugger

[OneCricketeer]

@niqdev I'm not too familiar with SBT plugins myself. I primarily stick to Maven&Gradle

[amitsahu07]

You can try using the below script. This will migrate you schema cross-cluster schema registry service.

schema_registry_source='http://hostname:8081'
schema_registry_sink='http://hostname2:8081'
TOPIC=topicname
schemapayload=curl -s $schema_registry_source/subjects/$TOPIC-value/versions/latest > /tmp/temp.txt && cat /tmp/temp.txt | awk -F "schema\":" '{print $2}' | sed 's/.$//'

curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" -d"{"schema":$schemapayload}" $schema_registry_sink/subjects/$TOPIC-value/versions

[jdang67]

Looks like export and import feature omit the schema Id and version number. How this supposed to work in this case: #1236 ?

thanks,

Jdang

[OneCricketeer]

@jdang67 Last I checked, it's not possible to upload a schema to a particular ID.