Skip to content

Commit

Permalink
[CARBONDATA-2369] updated the document about AVRO to carbon schema co…
Browse files Browse the repository at this point in the history
…nverter

updated the document about AVRO to carbon schema converter

This closes #2296
  • Loading branch information
ajantha-bhat authored and kumarvishal09 committed May 11, 2018
1 parent d85fb72 commit 443b717
Show file tree
Hide file tree
Showing 2 changed files with 19 additions and 7 deletions.
1 change: 1 addition & 0 deletions README.md
Expand Up @@ -43,6 +43,7 @@ CarbonData is built using Apache Maven, to [build CarbonData](https://github.com
* [Cluster Installation and Deployment](https://github.com/apache/carbondata/blob/master/docs/installation-guide.md)
* [Configuring Carbondata](https://github.com/apache/carbondata/blob/master/docs/configuration-parameters.md)
* [Streaming Ingestion](https://github.com/apache/carbondata/blob/master/docs/streaming-guide.md)
* [SDK Writer Guide](https://github.com/apache/carbondata/blob/master/docs/sdk-writer-guide.md)
* [CarbonData Pre-aggregate DataMap](https://github.com/apache/carbondata/blob/master/docs/datamap/preaggregate-datamap-guide.md)
* [CarbonData Timeseries DataMap](https://github.com/apache/carbondata/blob/master/docs/datamap/timeseries-datamap-guide.md)
* [FAQ](https://github.com/apache/carbondata/blob/master/docs/faq.md)
Expand Down
25 changes: 18 additions & 7 deletions docs/sdk-writer-guide.md
Expand Up @@ -52,6 +52,7 @@ import java.io.IOException;

import org.apache.carbondata.common.exceptions.sql.InvalidLoadOptionException;
import org.apache.carbondata.core.metadata.datatype.DataTypes;
import org.apache.carbondata.sdk.file.AvroCarbonWriter;
import org.apache.carbondata.sdk.file.CarbonWriter;
import org.apache.carbondata.sdk.file.Field;

Expand All @@ -75,25 +76,24 @@ public class TestSdkAvro {
" \"type\" : \"record\"," +
" \"name\" : \"Acme\"," +
" \"fields\" : ["
+ "{ \"name\" : \"name\", \"type\" : \"string\" },"
+ "{ \"name\" : \"fname\", \"type\" : \"string\" },"
+ "{ \"name\" : \"age\", \"type\" : \"int\" }]" +
"}";

String json = "{\"name\":\"bob\", \"age\":10}";
String json = "{\"fname\":\"bob\", \"age\":10}";

// conversion to GenericData.Record
JsonAvroConverter converter = new JsonAvroConverter();
GenericData.Record record = converter.convertToGenericDataRecord(
json.getBytes(CharEncoding.UTF_8), new org.apache.avro.Schema.Parser().parse(avroSchema));

// for sdk schema
Field[] fields = new Field[2];
fields[0] = new Field("name", DataTypes.STRING);
fields[1] = new Field("age", DataTypes.STRING);
// prepare carbon schema from avro schema
org.apache.carbondata.sdk.file.Schema carbonSchema =
AvroCarbonWriter.getCarbonSchemaFromAvroSchema(avroSchema);

try {
CarbonWriter writer = CarbonWriter.builder()
.withSchema(new org.apache.carbondata.sdk.file.Schema(fields))
.withSchema(carbonSchema)
.outputPath(path)
.buildWriterForAvroInput();

Expand Down Expand Up @@ -345,4 +345,15 @@ public Schema(Field[] fields);
* @return Schema
*/
public static Schema parseJson(String json);
```

### Class org.apache.carbondata.sdk.file.AvroCarbonWriter
```
/**
* converts avro schema to carbon schema, required by carbonWriter
*
* @param avroSchemaString json formatted avro schema as string
* @return carbon sdk schema
*/
public static org.apache.carbondata.sdk.file.Schema getCarbonSchemaFromAvroSchema(String avroSchemaString);
```

0 comments on commit 443b717

Please sign in to comment.