Adding co-occurrence metrics to facilitate finding related packages

[ruslandoga]

👋

Related ex_doc issue: elixir-lang/ex_doc#1811

Would it be possible to calculate and persist and make available through the API the co-occurrence metrics for packages from the fastly logs? For example, Phoenix and Ecto are downloaded together quite often and would probably have a high co-occurrence statistic and that would make it possible to prioritize ecto search results when something like "query" is searched from hexdocs.pm/phoenix

[ericmj]

I think we would have to log additional data to make that possible. Today the client doesn't send any information that would tell us that two requests are from the same mix deps.get other than possibly the IP address.

The first step would be to add a session identifier to requests made by https://github.com/hexpm/hex_core. This should probably be done by generating a UUID on hex_core start up and adding it as a header to all requests.

After that we can update the fastly logger to include the header in the log lines and update the stats collection job to check the header.

[ruslandoga]

Right, I was thinking about identifying the sessions as "requests made from the same IP address within a minute of each other". Having an actual session identifier would be great.

The first step would be to add a session identifier to requests made by https://github.com/hexpm/hex_core. This should probably be done by generating a UUID on hex_core start up and adding it as a header to all requests.

I'd be happy to work on this! I'll try to get something working by the end of the week.