Skip to content

Conversation

@alukach
Copy link
Contributor

@alukach alukach commented Sep 30, 2025

What I'm changing

Currently, some buckets/prefixes return parse errors while others do not:

The core issue regarding why the second API response failed to parse is that the youssef-harby account contains a populated metadata_public.domains array:

▶ curl -s https://source.coop/api/v1/products/youssef-harby/weather-station-realtime-parquet | jq .account.metadata_public.domains
[
  {
    "created_at": "2025-09-27T20:25:04.246Z",
    "domain": "youssefharby.com",
    "status": "unverified"
  }
]

Note that the account used in the successful API response lacks a populated metadata_public.domains array:

▶ curl -s https://source.coop/api/v1/products/harvard-lil/gov-data | jq .account.metadata_public.domains                   
[]

The proxy fails when it encounters a populated domain array due to the fact that the domain struct in the data proxy codebase does not match the API response. On the API, the status value is an enum1 added in S2:

export const AccountDomainSchema = z.object({
  domain: z.string(),
  status: z.enum(["unverified", "pending", "verified"]),
  // ...
});

However, the data proxy is expecting the legacy schema where a domain would merely have a boolean verified property:

#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct AccountDomain {
pub domain: String,
pub verified: bool,
}

How I did it

In this PR, we update the expected API schema to match the schema from within the source.coop codebase, namely within the following files:

I let Cursor AI take the wheel to update these types.

Also added tests with the real (but anonymized) data returned from https://source.coop/api/v1/products/youssef-harby/weather-station-realtime-parquet to ensure parsing would not throw an error.

Finally, improved the documentation of the file.

How to test it

  • See ce22b16, note that tests fail
  • See c9a9bbc, note that tests are now passing

PR Checklist

  • This PR has no breaking changes.
  • I have updated or added new tests to cover the changes in this PR.
  • This PR affects the Source Cooperative Frontend & API,
    and I have opened issue/PR #XXX to track the change.

Related Issues

Footnotes

  1. https://github.com/source-cooperative/source.coop/blob/c49aa11bb3f7a7bfd7caa40ff66193f9fe437b08/src/types/account.ts#L52

@yharby
Copy link

yharby commented Sep 30, 2025

@alukach Thank you so much 🙌

@alukach alukach force-pushed the fix/api-parsing-error branch 2 times, most recently from a000e72 to f497d48 Compare September 30, 2025 00:43
@alukach alukach marked this pull request as draft September 30, 2025 00:53
@alukach alukach force-pushed the fix/api-parsing-error branch from 55a6293 to ce22b16 Compare September 30, 2025 03:20
@alukach alukach marked this pull request as ready for review September 30, 2025 03:22
Copy link

@jedsundwall jedsundwall left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for jumping on this one! Weird!

@alukach alukach merged commit 6dec35d into main Sep 30, 2025
8 checks passed
@alukach alukach deleted the fix/api-parsing-error branch September 30, 2025 04:27
alukach pushed a commit that referenced this pull request Sep 30, 2025
🤖 I have created a release *beep* *boop*
---


##
[1.0.1](v1.0.0...v1.0.1)
(2025-09-30)


### Bug Fixes

* update source api types to match S2 codebase
([#90](#90))
([6dec35d](6dec35d))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: source-coop-release[bot] <187876225+source-coop-release[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants