Upgrade nixpkgs #3155

wolfgangwalther · 2024-01-11T18:22:53Z

I tried updating nixpkgs a few day ago. This didn't work out well, because the nixpkgs-unstable channel had some packages that we require broken. Then I wondered: Why do we even have nixpkgs-unstable in postgrest-nixpkgs-upgrade? So I changed that to use the latest stable channel - and then ran the upgrade again.

This time, I was able to make it work with a few changes here and there. One major change:

fuzzyset was marked as broken, so I replaced that.
fuzzyset requires text >=2.0.2 - which is only available in GHC 9.4+
Therefore, I upgraded to GHC 9.4.

That means that because of the fuzzyset dependency, we can't build with GHC 9.2 or GHC 9.0 anymore. I don't think that this is a problem... but let's wait and see what the Build-Cabal-Arm job says about that.

TODO:

Update stack

Resolves #3107
Supersedes #3112

Probably still blocked by #2834

wolfgangwalther · 2024-01-12T07:14:14Z

No idea why the nix build succeeds locally, but fails in CI. Shouldn't this be reproducible?

steve-chavez · 2024-01-12T15:40:54Z

It's taking a while to build this locally.. maybe the CI failure could be related to the Prepopulate Nix cache for Linux runners job? Perhaps opening another PR could confirm this?

develop7 · 2024-01-12T15:54:17Z

No idea why the nix build succeeds locally, but fails in CI. Shouldn't this be reproducible?

Could the local environment leak into the build? Does it build in nix-shell --pure?

steve-chavez · 2024-01-12T16:11:37Z

maybe the CI failure could be related to the Prepopulate Nix cache for Linux runners job? Perhaps opening another PR could confirm this?

Nope, just tried running the tests locally and failed too:

[nix-shell:~/Projects/postgrest]$ postgrest-test-spec
WARNING:  cast will be ignored because the source data type is a domain
psql:test/spec/fixtures/schema.sql:3158: WARNING:  cast will be ignored because the source data type is a domain
psql:test/spec/fixtures/schema.sql:3159: WARNING:  cast will be ignored because the target data type is a domain
psql:test/spec/fixtures/schema.sql:3160: WARNING:  cast will be ignored because the target data type is a domain
psql:test/spec/fixtures/schema.sql:3177: WARNING:  cast will be ignored because the source data type is a domain
psql:test/spec/fixtures/schema.sql:3178: WARNING:  cast will be ignored because the target data type is a domain
psql:test/spec/fixtures/schema.sql:3199: WARNING:  cast will be ignored because the source data type is a domain
psql:test/spec/fixtures/schema.sql:3200: WARNING:  cast will be ignored because the target data type is a domain
psql:test/spec/fixtures/schema.sql:3201: WARNING:  cast will be ignored because the target data type is a domain
psql:test/spec/fixtures/schema.sql:3220: WARNING:  cast will be ignored because the source data type is a domain
psql:test/spec/fixtures/schema.sql:3221: WARNING:  cast will be ignored because the target data type is a domain
psql:test/spec/fixtures/schema.sql:3222: WARNING:  cast will be ignored because the target data type is a domain
psql:test/spec/fixtures/schema.sql:3239: WARNING:  cast will be ignored because the source data type is a domain
psql:test/spec/fixtures/schema.sql:3240: WARNING:  cast will be ignored because the target data type is a domain
psql:test/spec/fixtures/schema.sql:3241: WARNING:  cast will be ignored because the target data type is a domain
postgrest-with-postgresql-16: You can connect with: psql 'postgres:///postgres?host=/run/user/1000/postgrest/postgrest-with-postgresql-16-sEF/socket' -U postgres
postgrest-with-postgresql-16: You can tail the logs with: tail -f /run/user/1000/postgrest/postgrest-with-postgresql-16-sEF/db.log
Warning: The package list for 'hackage.haskell.org' does not exist. Run 'cabal
update' to download it.
Warning: Requested index-state 2023-10-13T13:54:33Z is newer than
'hackage.haskell.org'! Falling back to older state ().
Resolving dependencies...
Error: cabal: Cannot run the test suite 'spec' because the solver did not find
a plan that included the test suites for postgrest-11.2.0. It is probably
worth trying again with test suites explicitly enabled in the configuration in
the cabal.project{.local} file. This will ask the solver to find a plan with
the test suites available. It will either fail with an explanation or find a
different plan that uses different versions of some other packages. Use the
'--dry-run' flag to see package versions and check that you are happy with the
choices.

Temporary directory kept at: /run/user/1000/postgrest/postgrest-with-postgresql-16-sEF

Same error message as CI

steve-chavez · 2024-01-12T16:12:59Z

Only the tests though, running the executable works fine:

[nix-shell:~/Projects/postgrest]$ postgrest-with-postgresql-15 postgrest-run
psql:test/spec/fixtures/schema.sql:2819: WARNING:  cast will be ignored because the source data type is a domain
psql:test/spec/fixtures/schema.sql:3158: WARNING:  cast will be ignored because the source data type is a domain
psql:test/spec/fixtures/schema.sql:3159: WARNING:  cast will be ignored because the target data type is a domain
psql:test/spec/fixtures/schema.sql:3160: WARNING:  cast will be ignored because the target data type is a domain
psql:test/spec/fixtures/schema.sql:3177: WARNING:  cast will be ignored because the source data type is a domain
psql:test/spec/fixtures/schema.sql:3178: WARNING:  cast will be ignored because the target data type is a domain
psql:test/spec/fixtures/schema.sql:3199: WARNING:  cast will be ignored because the source data type is a domain
psql:test/spec/fixtures/schema.sql:3200: WARNING:  cast will be ignored because the target data type is a domain
psql:test/spec/fixtures/schema.sql:3201: WARNING:  cast will be ignored because the target data type is a domain
psql:test/spec/fixtures/schema.sql:3220: WARNING:  cast will be ignored because the source data type is a domain
psql:test/spec/fixtures/schema.sql:3221: WARNING:  cast will be ignored because the target data type is a domain
psql:test/spec/fixtures/schema.sql:3222: WARNING:  cast will be ignored because the target data type is a domain
psql:test/spec/fixtures/schema.sql:3239: WARNING:  cast will be ignored because the source data type is a domain
psql:test/spec/fixtures/schema.sql:3240: WARNING:  cast will be ignored because the target data type is a domain
psql:test/spec/fixtures/schema.sql:3241: WARNING:  cast will be ignored because the target data type is a domain
postgrest-with-postgresql-15: You can connect with: psql 'postgres:///postgres?host=/run/user/1000/postgrest/postgrest-with-postgresql-15-v6T/socket' -U postgres
postgrest-with-postgresql-15: You can tail the logs with: tail -f /run/user/1000/postgrest/postgrest-with-postgresql-15-v6T/db.log
12/Jan/2024:11:12:22 -0500: Starting PostgREST 11.2.0 (bac81f0)...
12/Jan/2024:11:12:22 -0500: Attempting to connect to the database...
12/Jan/2024:11:12:22 -0500: Connection successful
12/Jan/2024:11:12:22 -0500: Listening on port 3000
12/Jan/2024:11:12:22 -0500: Listening for notifications on the pgrst channel
12/Jan/2024:11:12:22 -0500: Config reloaded
12/Jan/2024:11:12:22 -0500: Schema cache loaded

steve-chavez · 2024-01-12T16:15:21Z

Could the local environment leak into the build? Does it build in nix-shell --pure?

nix-shell --pure and postgrest-test-spec also fails for me.

steve-chavez · 2024-01-12T16:16:49Z

Probably still blocked by #2834

No problem with the above from my side. Let's just upgrade and figure it out later.

wolfgangwalther · 2024-01-12T18:54:42Z

I'm not sure what to conclude from the observations here. postgrest-test-spec is

failing in CI (nix-env -iA)
passing for me (nix-env -iA)
passing for me (nix-shell)
passing for me (nix-shell pure)
failing for Steve (nix-shell)
passing for Steve (nix-shell pure)

Correct?

I wonder what I need to do to make it fail.. so I can actually debug it.

wolfgangwalther · 2024-01-12T19:00:18Z

I wonder what I need to do to make it fail.. so I can actually debug it.

Aha. I deleted my ~/.cabal directory. And now it's failing for me, too.

So nix-shell --pure does not protect from that.

wolfgangwalther · 2024-01-12T19:02:26Z

Doing a cabal update makes it work again...

... but this is all confusing. We should not need to run cabal update - the nix-shell environment should provide all the packages we need. Clearly it doesn't, yet.

steve-chavez · 2024-01-12T19:13:06Z

passing for Steve (nix-shell pure)
Correct?

My bad, postgrest-test-spec also fails for me with nix-shell --pure. (somehow switched branches when trying that)

cabal.project

stack.yaml

wolfgangwalther · 2024-01-14T14:51:39Z

Added to commits to test building with GHC 9.6.x and GHC 9.8.x just out of curiosity..

wolfgangwalther · 2024-01-15T17:13:16Z

Added to commits to test building with GHC 9.6.x and GHC 9.8.x just out of curiosity..

Seems like our dependencies are not ready for GHC 9.6+, yet. Will drop those for now.

Previously this command upgraded to the latest unstable version of nixpkgs, but this was often broken. Taking the latest stable branch should give better results.

develop7 · 2024-01-16T17:28:34Z

Couldn't help but notice "prepopulate nix cache" job is building the xgcc from scratch (https://github.com/PostgREST/postgrest/actions/runs/7534062527/job/20507765748?pr=3155#step:4:8232) taking far too much time. Any idea why?

wolfgangwalther · 2024-01-16T17:33:12Z

Couldn't help but notice "prepopulate nix cache" job is building the xgcc from scratch (https://github.com/PostgREST/postgrest/actions/runs/7534062527/job/20507765748?pr=3155#step:4:8232) taking far too much time. Any idea why?

It's expected that the toolchain needs to be rebuilt after a nixpkgs upgrade, I think. I have not looked at the prepopulate nix cache job and the related changes in the last months, so I can't tell whether it does something odd there. But a complete rebuild after a nixpkgs update often takes a lot of time - and in the small machines that are run in CI, 5+ hours are expected for me.

Am I missing something?

steve-chavez

LGTM.

develop7 · 2024-01-18T12:29:37Z

and in the small machines that are run in CI, 5+ hours are expected for me

@wolfgangwalther I see. Are you suggesting these long-building outputs eventually make their way to PostgREST's Cachix? Fine by me, then.

wolfgangwalther · 2024-01-18T17:54:25Z

Are you suggesting these long-building outputs eventually make their way to PostgREST's Cachix?

That's my understanding, yes. Once those jobs run on the main branch, they will actually push to cachix.

wolfgangwalther commented Jan 14, 2024

View reviewed changes

cabal.project Show resolved Hide resolved

wolfgangwalther commented Jan 14, 2024

View reviewed changes

stack.yaml Outdated Show resolved Hide resolved

wolfgangwalther force-pushed the upgrade-nixpkgs branch from a63d771 to f53ad14 Compare January 14, 2024 14:50

wolfgangwalther force-pushed the upgrade-nixpkgs branch from f53ad14 to 846f12c Compare January 15, 2024 17:14

wolfgangwalther added 2 commits January 15, 2024 22:13

nix: Make postgrest-nixpkgs-upgrade take the latest stable version

6afb203

Previously this command upgraded to the latest unstable version of nixpkgs, but this was often broken. Taking the latest stable branch should give better results.

nix: Update nixpkgs and build with GHC 9.4.8

17e5faa

wolfgangwalther force-pushed the upgrade-nixpkgs branch from 846f12c to 17e5faa Compare January 15, 2024 21:14

steve-chavez approved these changes Jan 17, 2024

View reviewed changes

wolfgangwalther merged commit 6682bae into PostgREST:main Jan 18, 2024
30 of 31 checks passed

wolfgangwalther deleted the upgrade-nixpkgs branch January 18, 2024 17:54

This was referenced Jan 18, 2024

Test ghc 9.4 #3112

Closed

ci: Build with GHC 9.6.4 and 9.8.1 via cabal #3177

Merged

wolfgangwalther mentioned this pull request Feb 7, 2024

ci: Improve caching #3211

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade nixpkgs #3155

Upgrade nixpkgs #3155

wolfgangwalther commented Jan 11, 2024 •

edited

wolfgangwalther commented Jan 12, 2024

steve-chavez commented Jan 12, 2024

develop7 commented Jan 12, 2024

steve-chavez commented Jan 12, 2024

steve-chavez commented Jan 12, 2024

steve-chavez commented Jan 12, 2024 •

edited

steve-chavez commented Jan 12, 2024

wolfgangwalther commented Jan 12, 2024 •

edited

wolfgangwalther commented Jan 12, 2024

wolfgangwalther commented Jan 12, 2024

steve-chavez commented Jan 12, 2024

wolfgangwalther commented Jan 14, 2024

wolfgangwalther commented Jan 15, 2024

develop7 commented Jan 16, 2024

wolfgangwalther commented Jan 16, 2024

steve-chavez left a comment

develop7 commented Jan 18, 2024 •

edited

wolfgangwalther commented Jan 18, 2024

Upgrade nixpkgs #3155

Upgrade nixpkgs #3155

Conversation

wolfgangwalther commented Jan 11, 2024 • edited

wolfgangwalther commented Jan 12, 2024

steve-chavez commented Jan 12, 2024

develop7 commented Jan 12, 2024

steve-chavez commented Jan 12, 2024

steve-chavez commented Jan 12, 2024

steve-chavez commented Jan 12, 2024 • edited

steve-chavez commented Jan 12, 2024

wolfgangwalther commented Jan 12, 2024 • edited

wolfgangwalther commented Jan 12, 2024

wolfgangwalther commented Jan 12, 2024

steve-chavez commented Jan 12, 2024

wolfgangwalther commented Jan 14, 2024

wolfgangwalther commented Jan 15, 2024

develop7 commented Jan 16, 2024

wolfgangwalther commented Jan 16, 2024

steve-chavez left a comment

Choose a reason for hiding this comment

develop7 commented Jan 18, 2024 • edited

wolfgangwalther commented Jan 18, 2024

wolfgangwalther commented Jan 11, 2024 •

edited

steve-chavez commented Jan 12, 2024 •

edited

wolfgangwalther commented Jan 12, 2024 •

edited

develop7 commented Jan 18, 2024 •

edited