MP import feature only shows 3 #631

vnugent · 2022-12-22T21:20:41Z

Expected Behavior

MP user: https://www.mountainproject.com/user/200492669/andre-chiquito
OB profile: https://openbeta.io/u/all_terrain_human

The user has over 400 ticks but it seems only 3 are imported/shown

cc @Downster

musoke · 2023-03-12T19:36:55Z

I think I understand what happened: the importer couldn't handle the same route appearing twice in a row.

The same happened to my attempted import. The first 32 routes were fine, but the rest are missing. The 33rd tick in my logbook is of the same route as the 32nd.

Judging by the dates on this issue and their logbook, these were the three most recent routes at the time of the attempted import:

Left Of Time
The Twins Right
Hardman
Hardman
Mr. Aidman's Free
Black Hole

These are the three ticks imported successfully:

The Twins Right
Hardman
Left Of Time

It seems to have failed when it saw the second Hardman.

musoke · 2023-03-12T20:04:25Z

I suspect that this is related to the error message that appears when attempting to log multiple ticks on the same day through the web interface.

Is there a reason for that being disallowed? I often do a climb more than once in a day.

vnugent · 2023-04-18T14:16:13Z

Is there a reason for that being disallowed? I often do a climb more than once in a day.

I think due to date string being used as the unique constraint: OpenBeta/openbeta-graphql#175

musoke · 2023-04-18T14:19:52Z

Ah! Makes sense. Not sure I have the skills to fix it.

…

On Tue, Apr 18, 2023, 10:18 Viet Nguyen ***@***.***> wrote: Is there a reason for that being disallowed? I often do a climb more than once in a day. I think due to date string being used as the unique constraint: OpenBeta/openbeta-graphql#175 <OpenBeta/openbeta-graphql#175> — Reply to this email directly, view it on GitHub <#631 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AD7ET7BGL7OVPFGBUJ5EJITXB2PDVANCNFSM6AAAAAATHED22A> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

musoke · 2023-04-18T14:22:31Z

In any case, the importer should probably attempt to continue after encountering an error with individual ticks.

milbmr · 2023-04-29T22:44:28Z

I want to work at this issue but i need API keys to access users Profile.

vnugent · 2023-05-01T20:11:24Z

I want to work at this issue but i need API keys to access users Profile.

@milbmr
can you please send me an email: viet at openbeta dot io

musoke · 2023-07-13T12:15:13Z

I have another datapoint:

I imported the following log book:

It has two pairs out routes, each ticked twice on one day.

The first pair has different style.
The second pair has the same style.

Here's the list of ticks as reported by https://stg-api.openbeta.io:

{
  "data": {
    "userTicks": [
      {
        "_id": "64afe8808827f07ef7253506",
        "userId": "c26b3c91-bb05-42a9-8f8c-b2209c55781b",
        "name": "Lemonade",
        "notes": "",
        "climbId": "6cf437fa-027c-52ba-b0f0-7eda58fa4a2b",
        "style": "N/A",
        "attemptType": "N/A",
        "dateClimbed": 1682899200000,
        "grade": "V5",
        "source": "MP"
      },
      {
        "_id": "64afe8808827f07ef7253507",
        "userId": "c26b3c91-bb05-42a9-8f8c-b2209c55781b",
        "name": "Lemonade",
        "notes": "",
        "climbId": "6cf437fa-027c-52ba-b0f0-7eda58fa4a2b",
        "style": "Attempt",
        "attemptType": "Attempt",
        "dateClimbed": 1682899200000,
        "grade": "V5",
        "source": "MP"
      },
      {
        "_id": "64afe8808827f07ef7253508",
        "userId": "c26b3c91-bb05-42a9-8f8c-b2209c55781b",
        "name": "Cream Cheese",
        "notes": "",
        "climbId": "921a5290-a42e-5aec-8c67-f8d9fe4b01c8",
        "style": "Flash",
        "attemptType": "Flash",
        "dateClimbed": 1680307200000,
        "grade": "V6",
        "source": "MP"
      }
    ]
  }
}

The first pair imports successfully, the second does not.
It would seem that it is the combination of same day and style that fails. Differentiating by "notes" does not help.

musoke · 2023-07-13T14:36:37Z

I get this error when doing the import on a local dev open-tacos instance with the prod backend.

Unhandled Runtime Error

ApolloError: E11000 duplicate key error collection: openbeta.ticks index: climbId_1_dateClimbed_1_style_1_userId_1_source_1 dup key: { climbId: "921a5290-a42e-5aec-8c67-f8d9fe4b01c8", dateClimbed: new Date(1680321600000), style: "Flash", userId: "c26b3c91-bb05-42a9-8f8c-b2209c55781b", source: "MP" }


Call Stack
ApolloError
node_modules/@apollo/client/errors/index.js (26:0)
QueryManager.prototype.mutate/</</</<
node_modules/@apollo/client/core/QueryManager.js (97:0)
both
node_modules/@apollo/client/utilities/observables/asyncMap.js (16:45)
then/<
node_modules/@apollo/client/utilities/observables/asyncMap.js (9:56)
then
node_modules/@apollo/client/utilities/observables/asyncMap.js (9:0)
makeCallback/<
node_modules/@apollo/client/utilities/observables/asyncMap.js (17:0)
notifySubscription
node_modules/zen-observable-ts/module.js (132:0)
onNotify
node_modules/zen-observable-ts/module.js (176:0)
next
node_modules/zen-observable-ts/module.js (225:0)
iterateObserversSafely/<
node_modules/@apollo/client/utilities/observables/iteration.js (4:49)
iterateObserversSafely
node_modules/@apollo/client/utilities/observables/iteration.js (4:0)
next
node_modules/@apollo/client/utilities/observables/Concast.js (25:42)
notifySubscription
node_modules/zen-observable-ts/module.js (132:0)
onNotify
node_modules/zen-observable-ts/module.js (176:0)
next
node_modules/zen-observable-ts/module.js (225:0)
next
node_modules/@apollo/client/link/error/index.js (29:0)
notifySubscription
node_modules/zen-observable-ts/module.js (132:0)
onNotify
node_modules/zen-observable-ts/module.js (176:0)
next
node_modules/zen-observable-ts/module.js (225:0)
readJsonBody/<
node_modules/@apollo/client/link/http/parseAndCheckHttpResponse.js (144:0)

vnugent · 2023-07-13T14:39:02Z

I see what's going on now. The database has a unique constraint on climb, date, style, user, source to prevent the same tick from being imported multiple times. We can think of some other way to address re-import use case.

TickSchema.index({ climbId: 1, dateClimbed: 1, style: 1, userId: 1, source: 1 }, { unique: true })

https://github.com/OpenBeta/openbeta-graphql/blob/develop/src/db/TickSchema.ts#L27

musoke · 2023-07-13T14:40:43Z

I think the current implementation deletes previously imported ticks before doing the new import, so one can probably disable this addition check for now? Double check this.

vnugent · 2023-07-13T17:53:11Z

You're right. The import deletes all previous ones. We can change the indexes to something like this:

TickSchema.index({ userId: 1 }) // ticksByUser()
TickSchema.index({ climbId: 1, userId: 1 }) // for ticksByUserIdAndClimb()

The schema enforces a unique index on {climbId, dateClimbed, style, userID, source}. This is meant to prevent duplicating ticks imported from mountain project. However, it also prevents logging such ticks when they are legitimate. The import procedure also deletes imported ticks before reimporting, so this was overly cautious. Change the schema so it does not enforce uniqueness in this way. Instead, add non-unique indexes on { userId } and { userId, climbId }. Add a db migration script. Remove climb uniqueness test. Fixes OpenBeta/open-tacos#631

vnugent added the bug Something isn't working label Dec 22, 2022

vnugent added this to the v0.7 milestone Dec 22, 2022

vnugent added the help wanted Extra attention is needed label Dec 22, 2022

vnugent mentioned this issue Jan 5, 2023

Milestone 7 #634

Open

25 tasks

vnugent mentioned this issue Mar 3, 2023

Milestone 8 #716

Open

21 tasks

vnugent added the Difficulty: Hard label Mar 23, 2023

musoke mentioned this issue Apr 18, 2023

Importing Ticks/Ticks Testing #525

Closed

vnugent mentioned this issue Apr 22, 2023

🌱 Start here ➜ Milestone 9 #780

Closed

19 tasks

musoke mentioned this issue Jul 13, 2023

warn user if tick import errors #913

Closed

musoke self-assigned this Jul 13, 2023

musoke mentioned this issue Jul 13, 2023

fix: Allow identical ticks on one day OpenBeta/openbeta-graphql#340

Merged

vnugent closed this as completed in OpenBeta/openbeta-graphql#340 Jul 13, 2023

musoke added the Component: Log Book label Aug 3, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MP import feature only shows 3 #631

MP import feature only shows 3 #631

vnugent commented Dec 22, 2022 •

edited

Loading

musoke commented Mar 12, 2023 •

edited

Loading

musoke commented Mar 12, 2023

vnugent commented Apr 18, 2023

musoke commented Apr 18, 2023 via email •

edited

Loading

musoke commented Apr 18, 2023

milbmr commented Apr 29, 2023

vnugent commented May 1, 2023 •

edited

Loading

musoke commented Jul 13, 2023

musoke commented Jul 13, 2023 •

edited

Loading

vnugent commented Jul 13, 2023

musoke commented Jul 13, 2023 •

edited

Loading

vnugent commented Jul 13, 2023 •

edited

Loading

MP import feature only shows 3 #631

MP import feature only shows 3 #631

Comments

vnugent commented Dec 22, 2022 • edited Loading

Expected Behavior

musoke commented Mar 12, 2023 • edited Loading

musoke commented Mar 12, 2023

vnugent commented Apr 18, 2023

musoke commented Apr 18, 2023 via email • edited Loading

musoke commented Apr 18, 2023

milbmr commented Apr 29, 2023

vnugent commented May 1, 2023 • edited Loading

musoke commented Jul 13, 2023

musoke commented Jul 13, 2023 • edited Loading

vnugent commented Jul 13, 2023

musoke commented Jul 13, 2023 • edited Loading

vnugent commented Jul 13, 2023 • edited Loading

vnugent commented Dec 22, 2022 •

edited

Loading

musoke commented Mar 12, 2023 •

edited

Loading

musoke commented Apr 18, 2023 via email •

edited

Loading

vnugent commented May 1, 2023 •

edited

Loading

musoke commented Jul 13, 2023 •

edited

Loading

musoke commented Jul 13, 2023 •

edited

Loading

vnugent commented Jul 13, 2023 •

edited

Loading