Skip to content

Commit

Permalink
Remove some concurrency and performance bottlenecks in SpecificRecord…
Browse files Browse the repository at this point in the history
…Generator (#531)

* Make java file write wait by using reduce instead of count

* Remove unused pool

* Cleanup unused imports

* Make some shared global objects thread safe for concurrent access/modification

* Remove some concurrency and performance bottlenecks in SpecificRecordGenerator

---------

Co-authored-by: Karthik Ramgopal <kramgopa@linkedin.com>
  • Loading branch information
karthikrg and li-kramgopa committed Jan 11, 2024
1 parent dfd1900 commit 39d884c
Show file tree
Hide file tree
Showing 3 changed files with 75 additions and 134 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -250,9 +250,9 @@ public void run(OperationContext opContext) throws Exception {
throw new IllegalStateException("unable to create output folder " + outputFolder);
}
final Path outputDirectoryPath = outputFolder.toPath();
final SpecificRecordClassGenerator generator = new SpecificRecordClassGenerator();

int totalGeneratedClasses = allNamedSchemaList.parallelStream().map(allNamedSchemas -> {
SpecificRecordClassGenerator generator = new SpecificRecordClassGenerator();
HashSet<String> alreadyGeneratedSchemaNames = new HashSet<>();
List<JavaFile> generatedSpecificClasses = new ArrayList<>(allNamedSchemas.size());
for (AvroNamedSchema namedSchema : allNamedSchemas) {
Expand Down
Loading

0 comments on commit 39d884c

Please sign in to comment.