Upgrade Feast dependencies

[Wirick]

Context: I'm working on an integration with bigtable as an online store. I ran into problems with zip files not being supported because bigtable hbase is a large module. When I update spring boot there were a number of other cascading dependency issues that needed to be resolved. Currently this will allow for the work in #833 to be discussed separately from the dependency updates

This induces upgrades for a number of different packages in order to be
compatible, including kafka and the jakarta validation-api
Resolved logging errors in core/serving from seeing different logger
classes in the classpath by excluding them
I am able to build and run feast core and feast serving, along with bigquery batch ingestion with these updated dependencies.
UPDATE: I can also run my bigtable serving integration with these dependencies

However there was an upgrade needed in how we mock the url for the test database. Context: https://stackoverflow.com/questions/49088847/after-spring-boot-2-0-migration-jdbcurl-is-required-with-driverclassname

My original PR #851 had formatting changes so I am closing in favor of this pr.

Edit from @woop:
I have included more version bumps of related dependencies because it's too difficult to incrementally upgrade them one at a time. The following packages have been upgraded.

1. Spring Boot -> 2.3.1.RELEASE
2. gRPC -> 1.30.2
3. Protoc/Protobuf -> 3.12.2
4. Hibernate -> 5.4.18.Final
5. gRPC starter (devh, not lognet) -> 2.9.0.RELEASE
6. Apache Beam -> 2.22.0
7. Google Cloud Libraries -> 1.111.1
8. Kafka -> 2.5.0
9. OpenCensus -> 0.26.0

[feast-ci-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Wirick
To complete the pull request process, please assign zhilingc
You can assign the PR to them by writing /assign @zhilingc in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[feast-ci-bot]

Hi @Wirick. Thanks for your PR.

I'm waiting for a feast-dev member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ches]

Thanks @Wirick for the cleaned up PR, and sorry for the lecture on #851 about Spring bill of materials, from hasty reading on a small screen I had thought that and #793 were from the same contributor. We can take that matter up separately I think.

Does Spring Boot 2.3.1 bring any headache? Appears to be the latest stable. There are a few test errors too, looks like trying to use H2 but Postgres driver is activating. Otherwise the upgrade is good by me overall.

I ran into problems with zip files not being supported because bigtable hbase is a large module.

Ah yes, I've been there… spring-projects/spring-boot#16091. Upgrading Spring Boot is one way to fix it, and I think Feast is due for the upgrade anyway if it didn't cause too much hell where graphs intersect with Beam, gRPC, etc. ecosystems…

It brings up a related matter we've touched on on another channel—packaging Core becomes rather bloated as more storage engine support is added to Ingestion, since Core depends on Ingestion—but that cuts so deep that it needs a new issue, extending from #529 and its friends.

[Wirick]

@ches Thanks for the review. The failing tests are something that I tried to debug with no luck so far. the test url jdbc:h2:mem:testdb doesn't connect correctly in the integration tests but my connection to a real database in my integration tests works fine, but I haven't figured out what I'm doing wrong (So far tried using the h2 driver, switching the url property to jdbc-url and the postgres driver, etc, but no luck) Would love if anyone knew what to do

[Wirick]

Does Spring Boot 2.3.1 bring any headache? Appears to be the latest stable. There are a few test errors too, looks like trying to use H2 but Postgres driver is activating. Otherwise the upgrade is good by me overall.

I will 2.3.1 and see if I run into any issues

[woop]

/ok-to-test

[woop]

@Wirick This version bump requires more discussion because of the introduction of our Flyway migration scripts. I have managed to get the driver issues resolved on this branch locally, but then I start to encounter migration problems due to the incompatibility of PostreSQL and H2 database standards.

The following PostreSQL commands are not functioning in H2

• Most of the SET statements dont work
• ONLY doesnt work as part of an ALTER
• CREATE INDEX doesn't work for whatever reason
• INSERT INTO isn't supported

So we have three options

1. Try to refactor the migration scripts to see if we can use the lowest common denominator in SQL between H2 and PostgreSQL. This seems quite unlikely to be successful.
2. Having database specific flyway migrations
3. Remove H2 testing as part of unit tests and run these tests with Postgres as integration tests

From what I have seen so far it might be a bit tricky to support H2 here, and I am wary of trying to accommodate conditional or layered migrations. The overhead would be quite high. My preference is to run these tests only with Postgres as integration tests. I believe this is something @pyalex wanted to work towards in any case.

I'm not sure how that sits with teams that are not using Postgres though. @ches I believe you are still using MS SQL?

[woop]

Ok so I have decided to include more version bumps in this PR. It's very hard to bump versions of only one package since they are all inter-related, not to mention that these versions touch the new auth modules and our tests. I will edit the opening description.

[woop]

/test test-end-to-end-redis-cluster

[woop]

Ok so I have tried to clear as many issues as possible. Currently I am stuck with the spec stream tests not passing. At first I thought this was related to a dependency issue in Kafka/Zookeeper, but the exceptions are the same even if I run a local Kafka or Zookeeper.

It seems like the Beam pipeline never completes and just gets an interrupted exception, possibly related to acks not being received.

[pyalex]

From what I have seen so far it might be a bit tricky to support H2 here, and I am wary of trying to accommodate conditional or layered migrations. The overhead would be quite high. My preference is to run these tests only with Postgres as integration tests. I believe this is something @pyalex wanted to work towards in any case.

Yep, I started migration of big chunk of tests (in core) to integration tests with real postgres backend (via testcontainers). And that was motivated by significant amount of bugs in current release not catched by our current tests: 1. unit tests are based on our expectations on how the whole model layer would behave and those expectation are not met too often. 2. e2e tests are mostly covering only success cases.

My feeling is that though h2 may partly fix issue with tests it still won't solve it completely, because in production it still will be different database. And supporting database specific migration seems like excessive effort (though I was not aware that someone use in prod something other than Postgres when flyway was added). And is there any benefits to use h2 over postgres in tests except for maybe faster setup (which apparently run once)?

[Wirick]

Sorry wasn't at a keyboard for the weekend, I will take a look at this updated pr and see if I can make progress

[Wirick]

@woop When I execute make build-java I get this warning

[WARNING] Could not find artifact io.confluent:kafka-schema-registry-client:jar:5.3.2 in central (https://repo.maven.apache.org/maven2)

Try downloading the file manually from the project website.

Then, install it using the command:
    mvn install:install-file -DgroupId=io.confluent -DartifactId=kafka-schema-registry-client -Dversion=5.3.2 -Dpackaging=jar -Dfile=/path/to/file

Alternatively, if you host your own repository you can deploy the file there:
    mvn deploy:deploy-file -DgroupId=io.confluent -DartifactId=kafka-schema-registry-client -Dversion=5.3.2 -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]

Path to dependency:
        1) dev.feast:feast-ingestion:jar:0.6-SNAPSHOT
        2) org.apache.beam:beam-sdks-java-io-kafka:jar:2.22.0
        3) io.confluent:kafka-avro-serializer:jar:5.3.2
        4) io.confluent:kafka-schema-registry-client:jar:5.3.2


  io.confluent:kafka-schema-registry-client:jar:5.3.2

from the specified remote repositories:
  central (https://repo.maven.apache.org/maven2, releases=true, snapshots=false)

Downloading from central: https://repo.maven.apache.org/maven2/io/confluent/common-config/5.3.2/common-config-5.3.2.jar
[WARNING] Could not find artifact io.confluent:common-config:jar:5.3.2 in central (https://repo.maven.apache.org/maven2)

Try downloading the file manually from the project website.

I have encountered previously and was assuming that's why there was a problem with acking specs. I'm not sure how to make this error go away and where to add the repo, I feel like I've tried a couple different poms

[pyalex]

/test test-end-to-end-auth

[pyalex]

/test test-end-to-end-redis-cluster

[feast-ci-bot]

@Wirick: The following tests failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
test-end-to-end-redis-cluster	9912990	link	/test test-end-to-end-redis-cluster
test-end-to-end-auth	9912990	link	/test test-end-to-end-auth

Full PR test history

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[pyalex]

Resolved by #876