Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AVRO-3482: Reuse MAGIC in DataFileReader #1639

Merged
merged 1 commit into from Apr 27, 2022
Merged

AVRO-3482: Reuse MAGIC in DataFileReader #1639

merged 1 commit into from Apr 27, 2022

Conversation

rbalamohan
Copy link
Contributor

DataFileReader reads magic information twice. seek(0) is invoked
twice due to this. In cloud object stores, seeking back to 0 will
cause it to fall back to "random IO policy". Example of this is
S3A connector for s3. This causes suboptimal reads in object stores.
Refactoring in the patch addresses this case by reusing MAGIC.

Jira

https://issues.apache.org/jira/browse/AVRO-3482

Tests

  • Existing test cases cover this refactoring.

Commits

Documentation

N/A

@github-actions github-actions bot added the Java Pull Requests for Java binding label Apr 12, 2022
opwvhk
opwvhk approved these changes Apr 12, 2022
View reviewed changes
Copy link
Contributor

@opwvhk opwvhk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice & simple!

Normally, I'd want the method in a different order (initializeWithMagic first), but this makes for a cleaner diff.

@rbalamohan
Copy link
Contributor Author

@opwvhk : Thank you for the review and approval. Can you please let me know if there is any additional process involved for merging this?

@opwvhk
Copy link
Contributor

opwvhk commented Apr 15, 2022

Hi @rbalamohan,
The "additional process" is approval & merge by a committer (I'm a contributor). I'm hoping my opinion carries some weight, but I really don't know.

thiru-mg
thiru-mg reviewed Apr 26, 2022
View reviewed changes
lang/java/avro/src/main/java/org/apache/avro/file/DataFileReader.java Outdated
this(sin, reader, false, null);
}

public DataFileReader(SeekableInput sin, DatumReader<D> reader, byte[] magic) throws IOException {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the magic is embedded in the file and nobody outside the class (let alone the package) is likely to have the magic, it is better to make this constructor private. If we come across a valid use case in the future, we can then make it public.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @thiru-mg . I have fixed this in recent commit.

lang/java/avro/src/main/java/org/apache/avro/file/DataFileStream.java Outdated
}

/** Initialize the stream by reading from its head. */
void initializeWithMagic(InputStream in, byte[] magic) throws IOException {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To my taste, I would have left this function named initialize. The reason is there is a pattern followed throughout to call the non-constructor initializers initialize(). But I wouldn't insist on it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @thiru-mg . I have fixed this in recent commit.

@rbalamohan
AVRO-3482: Reuse MAGIC in DataFileReader
6f25bf2 
DataFileReader reads magic information twice. seek(0) is invoked
twice due to this. In cloud object stores, seeking back to 0 will
cause it to fall back to "random IO policy". Example of this is
S3A connector for s3. This causes suboptimal reads in object stores.
Refactoring in the patch addresses this case by reusing MAGIC.
thiru-mg
thiru-mg reviewed Apr 26, 2022
View reviewed changes
lang/java/avro/src/main/java/org/apache/avro/file/DataFileStream.java
}

/** Initialize the stream by reading from its head. */
void initialize(InputStream in, byte[] magic) throws IOException {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm sorry I should have been more complete last time. Can we make these two initialize functions be private, too?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DataFileReader extends this class and making it "private" would cause issue in accessing this method from DataFileReader.

@thiru-mg thiru-mg merged commit 9dedb2b into apache:master Apr 27, 2022
thiru-mg pushed a commit that referenced this pull request May 2, 2022
@rbalamohan @thiru-aqfer
AVRO-3482: Reuse MAGIC in DataFileReader (#1639)
2839bb7 
DataFileReader reads magic information twice. seek(0) is invoked
twice due to this. In cloud object stores, seeking back to 0 will
cause it to fall back to "random IO policy". Example of this is
S3A connector for s3. This causes suboptimal reads in object stores.
Refactoring in the patch addresses this case by reusing MAGIC.
martin-g pushed a commit to jklamer/avro that referenced this pull request May 4, 2022
@rbalamohan @martin-g
AVRO-3482: Reuse MAGIC in DataFileReader (apache#1639)
b63eacb 
DataFileReader reads magic information twice. seek(0) is invoked
twice due to this. In cloud object stores, seeking back to 0 will
cause it to fall back to "random IO policy". Example of this is
S3A connector for s3. This causes suboptimal reads in object stores.
Refactoring in the patch addresses this case by reusing MAGIC.
martin-g added a commit that referenced this pull request May 4, 2022
@jklamer @martin-g
@KyleSchoonover @massada @zcsizmadia @yanivru @dependabot @kordos @rbalamohan @oandrew @thiru-mg
[AVRO-3506] [rust] Single object writer (#1672)
7ba9447 
* Encoer v1 with interop data

* unit tested

* fmt

* Interop tested

* uneed file

* remove bugs

* clippy

* fix README

* rat fix

* Update lang/rust/avro/src/writer.rs

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>

* Update lang/rust/avro/src/writer.rs

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>

* Update lang/rust/avro/src/writer.rs

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>

* Update lang/rust/avro/src/writer.rs

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>

* Update lang/rust/avro/src/writer.rs

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>

* PR changes

* static setup

* Specific rename and interop test in script

* typo

* AVRO-3492: Add support for deriving Schema::Record aliases (#1647)

* AVRO-3492: Add support for deriving Schema::Record aliases

Uses Darling's 'multiple' attribute feature.

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* AVRO-3492: Add a test case with multiple attributes with different values for 'alias' key

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* AVRO-3494: Rust: uncomment some tests which actually pass

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* AVRO-3494: Uncomment a test for recursive types (#1648)

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* AVRO-3492: Add logic to derive the aliases for Schema::Enum (#1649)

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* AVRO-3415: Add code coverage report support for csharp (#1565)

* AVRO-3360 Updated XML documentation

* Revert "AVRO-3360 Updated XML documentation"

This reverts commit b8601c0.

* AVRO-3415 Add code coverage report support for csharp

* Ignore Updates and package references

* Updated names

* Sorted packages alphabetically

* Mode ReportGenerator instructions for global.

* Update versions.props

* Remove path

* Updated tabbing

* Cleanup version.props

* Add missing settings from version.props

* Updated from tabs to 2 space indents

* Added command in code block

* Fix carriage return

* force carriage return

* Another carriage return

* Added longer path to report

Co-authored-by: Kyle T. Schoonover <Kyle.T.Schoonover@nordstrom.com>

* AVRO-3384: Define C# Coding Style Guidelines (#1534)

* AVRO-3360 Updated XML documentation

* Revert "AVRO-3360 Updated XML documentation"

This reverts commit b8601c0.

* AVRO-3384 Initial check in

* Formatting fix

* Additional formatting

* More formatting

* Added additional rule

* Completed new line rules

* Indentation preferences complete

* Updated header

* Additional formatting

* More formatting changes

* Added spacing options

* Updated wrap options

* Additional documentation for styling

* Updated notes

* Updated more

* Added var preferences and Expression-bodied member preferences

* Initial styling rules documented

* Updated naming rules to reflect Roslyn naming rules

* Added other styling rule callouts.

* Updated Readme

* Updated rule

* Add header template

* Microsoft has a bug for semicolon which makes this not work.

* Added license

* Added note about IDE0055

Co-authored-by: Kyle T. Schoonover <Kyle.T.Schoonover@nordstrom.com>

* AVRO-3424: Added support to parse string into Schema.Type (#1571)

* AVRO-3360 Updated XML documentation

* Revert "AVRO-3360 Updated XML documentation"

This reverts commit b8601c0.

* AVRO-3424 Created extension method for converting string into a Schema.Type enumeration

* Updated functionality

* Removed breaking code

* Updated remove quotes

* Removed if from tests

Co-authored-by: Kyle T. Schoonover <Kyle.T.Schoonover@nordstrom.com>

* AVRO-3003: Fully qualify enum default value in C# code gen (#1596)

* AVRO-3458: Added tests for GenericRecord (#1606)

* AVRO-3360 Updated XML documentation

* Revert "AVRO-3360 Updated XML documentation"

This reverts commit b8601c0.

* AVRO-3458 Added tests for GenericRecord

* Moved Schema to const

* using discard

* Empty

* Add license

Co-authored-by: Kyle T. Schoonover <Kyle.T.Schoonover@nordstrom.com>

* AVRO-2883: Fix namespace mapping (#1610)

* Remove unused package references

* Replace namespace in text schema

* Remove namespace mapping

* Add unit tests

* Match namespace mapping used in ticket

* Make ReplaceMappedNamespacesInSchema private

* Mark NamespaceMapping obsolete

Co-authored-by: Zoltan Csizmadia <CsizmadiaZ@valassis.com>

* AVRO-2211: SchemaBuilder equivalent or other means of schema creation (#1597)

* AVRO-2211: Support schema creation

* Add license info to new files

* Fix documentation for FixedSchema ctor

* Remove and sort using

* Add missing brackets and replace var with explicit type

* Fix exception type in case of parsing

* Rename field to follow conventions

* AVRO 2211: Inlining temporary variable in linq

* AVRO-2211: Change exception type and add missing documentations

* AVRO-2211: Fix RecordSchema to set the positions of it's fields, instead of verifying it

* AVRO-2211: Fix RecordSchema fields assignment when creation new RecordSchema

* AVRO-2211: Change constructors of schema classes to factory method

* AVRO-2211: Add unit tests for RecordSchema and EnumSchema

* :AVRO-2211: Remove whitespace

* :AVRO-2211: Add symbol names verification for EnumSchema

* AVRO-2211: Fix enum name validation

* AVRO-2211: Throw AvroException consistently

* AVRO-2211: Throw AvroException in RecrodSchema consistently

* AVRO-2211: Remove duplicate factory methods on MapSchema

* AVRO-2211: Remove redundant parameter doc

* AVRO-2211: Add Schema creation tests

* AVRO-2211: Change ValidateSymbol to throw exception

* AVRO-2211: Fix typo

* AVRO-2211: Fix code QL issues

* AVRO-2211: Fix typo

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>

* AVRO-3841: Try exact schema match first in union type (#1635)

* Try exact schema match

* Fix formatting

* Add tests for exception

Co-authored-by: Zoltan Csizmadia <CsizmadiaZ@valassis.com>

* AVRO-3495: Rust: Fields order should not matter (#1650)

* AVRO-3495: The order of the struct's fields and schema's fields should not matter

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* AVRO-3495: Use the lookup table when comparing values against fields by name

Until now it was expected that both the schema fields and the input
values are sorted the same way.

Use BTreeMap instead of HashMap for the lookup table because otherwise
the assertion on the validation error messages is impossible due to
random printing of the map's entries

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* AVRO-3495: Update the test case

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* Bump slf4j.version from 1.7.33 to 1.7.36 in /lang/java (#1646)

Bumps `slf4j.version` from 1.7.33 to 1.7.36.

Updates `slf4j-api` from 1.7.33 to 1.7.36
- [Release notes](https://github.com/qos-ch/slf4j/releases)
- [Commits](qos-ch/slf4j@v_1.7.33...v_1.7.36)

Updates `slf4j-simple` from 1.7.33 to 1.7.36
- [Release notes](https://github.com/qos-ch/slf4j/releases)
- [Commits](qos-ch/slf4j@v_1.7.33...v_1.7.36)

Updates `slf4j-log4j12` from 1.7.33 to 1.7.36
- [Release notes](https://github.com/qos-ch/slf4j/releases)
- [Commits](qos-ch/slf4j@v_1.7.33...v_1.7.36)

---
updated-dependencies:
- dependency-name: org.slf4j:slf4j-api
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: org.slf4j:slf4j-simple
  dependency-type: direct:development
  update-type: version-update:semver-patch
- dependency-name: org.slf4j:slf4j-log4j12
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* AVRO-3491 Avoid a cast after is check (#1645)

* AVRO-3360 Updated XML documentation

* Revert "AVRO-3360 Updated XML documentation"

This reverts commit b8601c0.

* AVRO-3491 Avoid a cast after is check

Co-authored-by: Kyle T. Schoonover <Kyle.T.Schoonover@nordstrom.com>

* AVRO-3496: Rust: Use visitor.visit_borrowed_str() when possible (#1652)

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* AVRO-3477: Add unit tests for logical types with fixed base type (#1629)

* Support fixed base type for logical types

* Tweak

* Revert

* Fix fixed type definition

* Add AvroGen tests

Co-authored-by: Zoltan Csizmadia <CsizmadiaZ@valassis.com>

* AVRO-3465: Add avrogen protocol tests (#1616)

* Add avrogen protocol tests

* Add protocol test case

* Fix merge conflicts

Co-authored-by: Zoltan Csizmadia <CsizmadiaZ@valassis.com>

* AVRO-3484: Add support for deriving a default value for a record field (#1651)

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* AVRO-3497 Simplify conditional expression (#1658)

* AVRO-3497 Simplify conditional expression

* Added null check back

* Updated tests

* AVRO-3500: Use property-based testing for the IT tests in avro_derive module (#1659)

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* Configure Dependabot to check for Rust updates daily

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* AVRO-3501: Rust: Cache ~/.cargo and target folder for faster builds (#1661)

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* Avro 3502 logical type wrong order (#1664)

* AVRO-3501: Rust: Cache ~/.cargo and target folder for faster builds

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* AVRO-3502: Rust: Wrong [ORDER] for Parsing Canonical Form

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* Update uuid requirement from 0.8.2 to 1.0.0 in /lang/rust (#1660)

* Update uuid requirement from 0.8.2 to 1.0.0 in /lang/rust

Updates the requirements on [uuid](https://github.com/uuid-rs/uuid) to permit the latest version.
- [Release notes](https://github.com/uuid-rs/uuid/releases)
- [Commits](uuid-rs/uuid@0.8.2...1.0.0)

---
updated-dependencies:
- dependency-name: uuid
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* Issue #1660 - Fix compilation errors after updating uuid crate from 0.8 to 1.0

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* Bump jmh.version from 1.34 to 1.35 in /lang/java (#1662)

Bumps `jmh.version` from 1.34 to 1.35.

Updates `jmh-core` from 1.34 to 1.35

Updates `jmh-generator-annprocess` from 1.34 to 1.35

---
updated-dependencies:
- dependency-name: org.openjdk.jmh:jmh-core
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: org.openjdk.jmh:jmh-generator-annprocess
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump zstd-jni from 1.5.1-1 to 1.5.2-2 in /lang/java (#1663)

Bumps [zstd-jni](https://github.com/luben/zstd-jni) from 1.5.1-1 to 1.5.2-2.
- [Release notes](https://github.com/luben/zstd-jni/releases)
- [Commits](luben/zstd-jni@v1.5.1-1...v1.5.2-2)

---
updated-dependencies:
- dependency-name: com.github.luben:zstd-jni
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump libthrift from 0.15.0 to 0.16.0 in /lang/java (#1665)

Bumps [libthrift](https://github.com/apache/thrift) from 0.15.0 to 0.16.0.
- [Release notes](https://github.com/apache/thrift/releases)
- [Changelog](https://github.com/apache/thrift/blob/master/CHANGES.md)
- [Commits](apache/thrift@v0.15.0...v0.16.0)

---
updated-dependencies:
- dependency-name: org.apache.thrift:libthrift
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* AVRO-3498 Deprecate NameCtorKey (#1657)

* AVRO-3490 Updated to use throw expressions (#1644)

* AVRO-3360 Updated XML documentation

* Revert "AVRO-3360 Updated XML documentation"

This reverts commit b8601c0.

* AVRO-3490 Updated to use throw expressions

* Additional expressions

Co-authored-by: Kyle T. Schoonover <Kyle.T.Schoonover@nordstrom.com>

* Bump grpc.version from 1.45.0 to 1.45.1 in /lang/java (#1671)

Bumps `grpc.version` from 1.45.0 to 1.45.1.

Updates `grpc-core` from 1.45.0 to 1.45.1
- [Release notes](https://github.com/grpc/grpc-java/releases)
- [Commits](grpc/grpc-java@v1.45.0...v1.45.1)

Updates `grpc-stub` from 1.45.0 to 1.45.1
- [Release notes](https://github.com/grpc/grpc-java/releases)
- [Commits](grpc/grpc-java@v1.45.0...v1.45.1)

Updates `grpc-netty` from 1.45.0 to 1.45.1
- [Release notes](https://github.com/grpc/grpc-java/releases)
- [Commits](grpc/grpc-java@v1.45.0...v1.45.1)

---
updated-dependencies:
- dependency-name: io.grpc:grpc-core
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: io.grpc:grpc-stub
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: io.grpc:grpc-netty
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump underscore from 1.13.2 to 1.13.3 in /lang/js (#1669)

Bumps [underscore](https://github.com/jashkenas/underscore) from 1.13.2 to 1.13.3.
- [Release notes](https://github.com/jashkenas/underscore/releases)
- [Commits](jashkenas/underscore@1.13.2...1.13.3)

---
updated-dependencies:
- dependency-name: underscore
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* AVRO-3484: Followup Check default json parsing at compile time for derive macro  (#1668)

* check json parsing at compile time

* fmt

* AVRO-3427: skip creation of namespace directories for csharp schema (#1578)

* Add new argument parameter --skip-directories. It will skip creation of directories for namespace. Just generate classes in output directory

* Add missing doc param description

* Fix Unit tests after merge with master

* Fix Unit tests after merge with master

* C# Add unit tests for --skip-directories option

Co-authored-by: Pawel Kordowski <pawel.kordowski@stepstone.com>

* AVRO-3482: Reuse MAGIC in DataFileReader (#1639)

DataFileReader reads magic information twice. seek(0) is invoked
twice due to this. In cloud object stores, seeking back to 0 will
cause it to fall back to "random IO policy". Example of this is
S3A connector for s3. This causes suboptimal reads in object stores.
Refactoring in the patch addresses this case by reusing MAGIC.

* AVRO-2870: Avoid throwing from destructor in DataFileWriterBase (#921)

Co-authored-by: Thiruvalluvan M G <thiru@apache.org>

* Updated the checksum for PHP composer download (#1677)

* Remove trailing ^M to make Git happy

Related to: 72e1135

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* Encoer v1 with interop data

* unit tested

* fmt

* AVRO-3506: Cleanup and minor improvements

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* AVRO-3506: Cleanup

Give a better name to TestGenerateInteropSingleObjectEncoding
Remove useless lifetime in schema.rs
Remove .json files for the single object encoded test file

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* AVRO-3506: Add licence header to TestGenerateInteropSingleObjectEncoding

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* AVRO-3506: Fix spotless issues in the new Java test classes

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* AVRO-3506: Fix the path to the schema file

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* AVRO-3506: Fix the id to match the expected value

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* AVRO-3506: Fix spotless again

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>
Co-authored-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>
Co-authored-by: Kyle Schoonover <kyle@minmaxcorp.com>
Co-authored-by: Kyle T. Schoonover <Kyle.T.Schoonover@nordstrom.com>
Co-authored-by: Jose Massada <jose.massada@gmail.com>
Co-authored-by: Zoltan Csizmadia <zcsizmadia@gmail.com>
Co-authored-by: Zoltan Csizmadia <CsizmadiaZ@valassis.com>
Co-authored-by: yanivru <yaniv.ru@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kordos <pkordowsk@gmail.com>
Co-authored-by: Pawel Kordowski <pawel.kordowski@stepstone.com>
Co-authored-by: rbalamohan <rbalamohan@apache.org>
Co-authored-by: Andrew Onyshchuk <andryk.rv@gmail.com>
Co-authored-by: Thiruvalluvan M G <thiru@apache.org>
martin-g pushed a commit that referenced this pull request May 4, 2022
@jklamer @martin-g
[AVRO-3506] [rust] Single object writer (#1672)
d6f6b79 
* Encoer v1 with interop data

* unit tested

* fmt

* Interop tested

* uneed file

* remove bugs

* clippy

* fix README

* rat fix

* Update lang/rust/avro/src/writer.rs

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>

* Update lang/rust/avro/src/writer.rs

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>

* Update lang/rust/avro/src/writer.rs

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>

* Update lang/rust/avro/src/writer.rs

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>

* Update lang/rust/avro/src/writer.rs

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>

* PR changes

* static setup

* Specific rename and interop test in script

* typo

* AVRO-3492: Add support for deriving Schema::Record aliases (#1647)

* AVRO-3492: Add support for deriving Schema::Record aliases

Uses Darling's 'multiple' attribute feature.

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* AVRO-3492: Add a test case with multiple attributes with different values for 'alias' key

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* AVRO-3494: Rust: uncomment some tests which actually pass

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* AVRO-3494: Uncomment a test for recursive types (#1648)

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* AVRO-3492: Add logic to derive the aliases for Schema::Enum (#1649)

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* AVRO-3415: Add code coverage report support for csharp (#1565)

* AVRO-3360 Updated XML documentation

* Revert "AVRO-3360 Updated XML documentation"

This reverts commit b8601c0.

* AVRO-3415 Add code coverage report support for csharp

* Ignore Updates and package references

* Updated names

* Sorted packages alphabetically

* Mode ReportGenerator instructions for global.

* Update versions.props

* Remove path

* Updated tabbing

* Cleanup version.props

* Add missing settings from version.props

* Updated from tabs to 2 space indents

* Added command in code block

* Fix carriage return

* force carriage return

* Another carriage return

* Added longer path to report

Co-authored-by: Kyle T. Schoonover <Kyle.T.Schoonover@nordstrom.com>

* AVRO-3384: Define C# Coding Style Guidelines (#1534)

* AVRO-3360 Updated XML documentation

* Revert "AVRO-3360 Updated XML documentation"

This reverts commit b8601c0.

* AVRO-3384 Initial check in

* Formatting fix

* Additional formatting

* More formatting

* Added additional rule

* Completed new line rules

* Indentation preferences complete

* Updated header

* Additional formatting

* More formatting changes

* Added spacing options

* Updated wrap options

* Additional documentation for styling

* Updated notes

* Updated more

* Added var preferences and Expression-bodied member preferences

* Initial styling rules documented

* Updated naming rules to reflect Roslyn naming rules

* Added other styling rule callouts.

* Updated Readme

* Updated rule

* Add header template

* Microsoft has a bug for semicolon which makes this not work.

* Added license

* Added note about IDE0055

Co-authored-by: Kyle T. Schoonover <Kyle.T.Schoonover@nordstrom.com>

* AVRO-3424: Added support to parse string into Schema.Type (#1571)

* AVRO-3360 Updated XML documentation

* Revert "AVRO-3360 Updated XML documentation"

This reverts commit b8601c0.

* AVRO-3424 Created extension method for converting string into a Schema.Type enumeration

* Updated functionality

* Removed breaking code

* Updated remove quotes

* Removed if from tests

Co-authored-by: Kyle T. Schoonover <Kyle.T.Schoonover@nordstrom.com>

* AVRO-3003: Fully qualify enum default value in C# code gen (#1596)

* AVRO-3458: Added tests for GenericRecord (#1606)

* AVRO-3360 Updated XML documentation

* Revert "AVRO-3360 Updated XML documentation"

This reverts commit b8601c0.

* AVRO-3458 Added tests for GenericRecord

* Moved Schema to const

* using discard

* Empty

* Add license

Co-authored-by: Kyle T. Schoonover <Kyle.T.Schoonover@nordstrom.com>

* AVRO-2883: Fix namespace mapping (#1610)

* Remove unused package references

* Replace namespace in text schema

* Remove namespace mapping

* Add unit tests

* Match namespace mapping used in ticket

* Make ReplaceMappedNamespacesInSchema private

* Mark NamespaceMapping obsolete

Co-authored-by: Zoltan Csizmadia <CsizmadiaZ@valassis.com>

* AVRO-2211: SchemaBuilder equivalent or other means of schema creation (#1597)

* AVRO-2211: Support schema creation

* Add license info to new files

* Fix documentation for FixedSchema ctor

* Remove and sort using

* Add missing brackets and replace var with explicit type

* Fix exception type in case of parsing

* Rename field to follow conventions

* AVRO 2211: Inlining temporary variable in linq

* AVRO-2211: Change exception type and add missing documentations

* AVRO-2211: Fix RecordSchema to set the positions of it's fields, instead of verifying it

* AVRO-2211: Fix RecordSchema fields assignment when creation new RecordSchema

* AVRO-2211: Change constructors of schema classes to factory method

* AVRO-2211: Add unit tests for RecordSchema and EnumSchema

* :AVRO-2211: Remove whitespace

* :AVRO-2211: Add symbol names verification for EnumSchema

* AVRO-2211: Fix enum name validation

* AVRO-2211: Throw AvroException consistently

* AVRO-2211: Throw AvroException in RecrodSchema consistently

* AVRO-2211: Remove duplicate factory methods on MapSchema

* AVRO-2211: Remove redundant parameter doc

* AVRO-2211: Add Schema creation tests

* AVRO-2211: Change ValidateSymbol to throw exception

* AVRO-2211: Fix typo

* AVRO-2211: Fix code QL issues

* AVRO-2211: Fix typo

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>

* AVRO-3841: Try exact schema match first in union type (#1635)

* Try exact schema match

* Fix formatting

* Add tests for exception

Co-authored-by: Zoltan Csizmadia <CsizmadiaZ@valassis.com>

* AVRO-3495: Rust: Fields order should not matter (#1650)

* AVRO-3495: The order of the struct's fields and schema's fields should not matter

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* AVRO-3495: Use the lookup table when comparing values against fields by name

Until now it was expected that both the schema fields and the input
values are sorted the same way.

Use BTreeMap instead of HashMap for the lookup table because otherwise
the assertion on the validation error messages is impossible due to
random printing of the map's entries

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* AVRO-3495: Update the test case

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* Bump slf4j.version from 1.7.33 to 1.7.36 in /lang/java (#1646)

Bumps `slf4j.version` from 1.7.33 to 1.7.36.

Updates `slf4j-api` from 1.7.33 to 1.7.36
- [Release notes](https://github.com/qos-ch/slf4j/releases)
- [Commits](qos-ch/slf4j@v_1.7.33...v_1.7.36)

Updates `slf4j-simple` from 1.7.33 to 1.7.36
- [Release notes](https://github.com/qos-ch/slf4j/releases)
- [Commits](qos-ch/slf4j@v_1.7.33...v_1.7.36)

Updates `slf4j-log4j12` from 1.7.33 to 1.7.36
- [Release notes](https://github.com/qos-ch/slf4j/releases)
- [Commits](qos-ch/slf4j@v_1.7.33...v_1.7.36)

---
updated-dependencies:
- dependency-name: org.slf4j:slf4j-api
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: org.slf4j:slf4j-simple
  dependency-type: direct:development
  update-type: version-update:semver-patch
- dependency-name: org.slf4j:slf4j-log4j12
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* AVRO-3491 Avoid a cast after is check (#1645)

* AVRO-3360 Updated XML documentation

* Revert "AVRO-3360 Updated XML documentation"

This reverts commit b8601c0.

* AVRO-3491 Avoid a cast after is check

Co-authored-by: Kyle T. Schoonover <Kyle.T.Schoonover@nordstrom.com>

* AVRO-3496: Rust: Use visitor.visit_borrowed_str() when possible (#1652)

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* AVRO-3477: Add unit tests for logical types with fixed base type (#1629)

* Support fixed base type for logical types

* Tweak

* Revert

* Fix fixed type definition

* Add AvroGen tests

Co-authored-by: Zoltan Csizmadia <CsizmadiaZ@valassis.com>

* AVRO-3465: Add avrogen protocol tests (#1616)

* Add avrogen protocol tests

* Add protocol test case

* Fix merge conflicts

Co-authored-by: Zoltan Csizmadia <CsizmadiaZ@valassis.com>

* AVRO-3484: Add support for deriving a default value for a record field (#1651)

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* AVRO-3497 Simplify conditional expression (#1658)

* AVRO-3497 Simplify conditional expression

* Added null check back

* Updated tests

* AVRO-3500: Use property-based testing for the IT tests in avro_derive module (#1659)

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* Configure Dependabot to check for Rust updates daily

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* AVRO-3501: Rust: Cache ~/.cargo and target folder for faster builds (#1661)

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* Avro 3502 logical type wrong order (#1664)

* AVRO-3501: Rust: Cache ~/.cargo and target folder for faster builds

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* AVRO-3502: Rust: Wrong [ORDER] for Parsing Canonical Form

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* Update uuid requirement from 0.8.2 to 1.0.0 in /lang/rust (#1660)

* Update uuid requirement from 0.8.2 to 1.0.0 in /lang/rust

Updates the requirements on [uuid](https://github.com/uuid-rs/uuid) to permit the latest version.
- [Release notes](https://github.com/uuid-rs/uuid/releases)
- [Commits](uuid-rs/uuid@0.8.2...1.0.0)

---
updated-dependencies:
- dependency-name: uuid
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* Issue #1660 - Fix compilation errors after updating uuid crate from 0.8 to 1.0

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* Bump jmh.version from 1.34 to 1.35 in /lang/java (#1662)

Bumps `jmh.version` from 1.34 to 1.35.

Updates `jmh-core` from 1.34 to 1.35

Updates `jmh-generator-annprocess` from 1.34 to 1.35

---
updated-dependencies:
- dependency-name: org.openjdk.jmh:jmh-core
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: org.openjdk.jmh:jmh-generator-annprocess
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump zstd-jni from 1.5.1-1 to 1.5.2-2 in /lang/java (#1663)

Bumps [zstd-jni](https://github.com/luben/zstd-jni) from 1.5.1-1 to 1.5.2-2.
- [Release notes](https://github.com/luben/zstd-jni/releases)
- [Commits](luben/zstd-jni@v1.5.1-1...v1.5.2-2)

---
updated-dependencies:
- dependency-name: com.github.luben:zstd-jni
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump libthrift from 0.15.0 to 0.16.0 in /lang/java (#1665)

Bumps [libthrift](https://github.com/apache/thrift) from 0.15.0 to 0.16.0.
- [Release notes](https://github.com/apache/thrift/releases)
- [Changelog](https://github.com/apache/thrift/blob/master/CHANGES.md)
- [Commits](apache/thrift@v0.15.0...v0.16.0)

---
updated-dependencies:
- dependency-name: org.apache.thrift:libthrift
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* AVRO-3498 Deprecate NameCtorKey (#1657)

* AVRO-3490 Updated to use throw expressions (#1644)

* AVRO-3360 Updated XML documentation

* Revert "AVRO-3360 Updated XML documentation"

This reverts commit b8601c0.

* AVRO-3490 Updated to use throw expressions

* Additional expressions

Co-authored-by: Kyle T. Schoonover <Kyle.T.Schoonover@nordstrom.com>

* Bump grpc.version from 1.45.0 to 1.45.1 in /lang/java (#1671)

Bumps `grpc.version` from 1.45.0 to 1.45.1.

Updates `grpc-core` from 1.45.0 to 1.45.1
- [Release notes](https://github.com/grpc/grpc-java/releases)
- [Commits](grpc/grpc-java@v1.45.0...v1.45.1)

Updates `grpc-stub` from 1.45.0 to 1.45.1
- [Release notes](https://github.com/grpc/grpc-java/releases)
- [Commits](grpc/grpc-java@v1.45.0...v1.45.1)

Updates `grpc-netty` from 1.45.0 to 1.45.1
- [Release notes](https://github.com/grpc/grpc-java/releases)
- [Commits](grpc/grpc-java@v1.45.0...v1.45.1)

---
updated-dependencies:
- dependency-name: io.grpc:grpc-core
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: io.grpc:grpc-stub
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: io.grpc:grpc-netty
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump underscore from 1.13.2 to 1.13.3 in /lang/js (#1669)

Bumps [underscore](https://github.com/jashkenas/underscore) from 1.13.2 to 1.13.3.
- [Release notes](https://github.com/jashkenas/underscore/releases)
- [Commits](jashkenas/underscore@1.13.2...1.13.3)

---
updated-dependencies:
- dependency-name: underscore
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* AVRO-3484: Followup Check default json parsing at compile time for derive macro  (#1668)

* check json parsing at compile time

* fmt

* AVRO-3427: skip creation of namespace directories for csharp schema (#1578)

* Add new argument parameter --skip-directories. It will skip creation of directories for namespace. Just generate classes in output directory

* Add missing doc param description

* Fix Unit tests after merge with master

* Fix Unit tests after merge with master

* C# Add unit tests for --skip-directories option

Co-authored-by: Pawel Kordowski <pawel.kordowski@stepstone.com>

* AVRO-3482: Reuse MAGIC in DataFileReader (#1639)

DataFileReader reads magic information twice. seek(0) is invoked
twice due to this. In cloud object stores, seeking back to 0 will
cause it to fall back to "random IO policy". Example of this is
S3A connector for s3. This causes suboptimal reads in object stores.
Refactoring in the patch addresses this case by reusing MAGIC.

* AVRO-2870: Avoid throwing from destructor in DataFileWriterBase (#921)

Co-authored-by: Thiruvalluvan M G <thiru@apache.org>

* Updated the checksum for PHP composer download (#1677)

* Remove trailing ^M to make Git happy

Related to: 72e1135

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* Encoer v1 with interop data

* unit tested

* fmt

* AVRO-3506: Cleanup and minor improvements

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* AVRO-3506: Cleanup

Give a better name to TestGenerateInteropSingleObjectEncoding
Remove useless lifetime in schema.rs
Remove .json files for the single object encoded test file

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* AVRO-3506: Add licence header to TestGenerateInteropSingleObjectEncoding

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* AVRO-3506: Fix spotless issues in the new Java test classes

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* AVRO-3506: Fix the path to the schema file

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* AVRO-3506: Fix the id to match the expected value

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

* AVRO-3506: Fix spotless again

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>
Co-authored-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>
Co-authored-by: Kyle Schoonover <kyle@minmaxcorp.com>
Co-authored-by: Kyle T. Schoonover <Kyle.T.Schoonover@nordstrom.com>
Co-authored-by: Jose Massada <jose.massada@gmail.com>
Co-authored-by: Zoltan Csizmadia <zcsizmadia@gmail.com>
Co-authored-by: Zoltan Csizmadia <CsizmadiaZ@valassis.com>
Co-authored-by: yanivru <yaniv.ru@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kordos <pkordowsk@gmail.com>
Co-authored-by: Pawel Kordowski <pawel.kordowski@stepstone.com>
Co-authored-by: rbalamohan <rbalamohan@apache.org>
Co-authored-by: Andrew Onyshchuk <andryk.rv@gmail.com>
Co-authored-by: Thiruvalluvan M G <thiru@apache.org>
(cherry picked from commit 7ba9447)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@steveloughran steveloughran steveloughran left review comments

@opwvhk opwvhk opwvhk approved these changes

@thiru-mg thiru-mg thiru-mg approved these changes

Assignees
No one assigned
Labels
Java Pull Requests for Java binding
Projects
None yet
Milestone
No milestone
4 participants
@rbalamohan @opwvhk @steveloughran @thiru-mg