Skyperious 4.7 Online sync stopping at 4/2018

[RikuAotsuki]

I got the program to do the online sync, but for whatever reason it decides nothing exists before April 2018, when the database should go back to April 2017.

I used Microsoft's export too, and using their (awful) parser I can see that those chats are definitely there; Skyperious' sync just isn't going back that far.

As a secondary issue, Skyperious can't create a viewable database from the export .json or .tar; It parses it, says how many messages, and then when you click "Chats" to look there's nothing there. No chats, no messages.

[suurjaak]

Are the messages from 2017 visible in the official Skype web interface? If not, then they're just not available for live retrieval.

Secondary issue: confirmed, this is a new bug. Will be fixed in an upcoming release.

[RikuAotsuki]

They're visible in the program version of Skype and in the export... and yes, the browser version has it too.

Admittedly not the place but I'm not sure where to ask: in advanced options under the export chat template, how would I get it format files as Year-Month-Day Title? Can't seem to find what variables to use.

[suurjaak]

Can you do the following in Skyperious:

• log in to Skype on the Online-page
• open a chat that is lacking earlier messages
• open the Python console (menu Help -> Show Python console)
• execute the following, line by line:

    chat = db.live.skype.chats[live.identity_to_id(page.chat["identity"])]
    for msg in chat.getMsgs(): print msg.time

• execute the last line over and over again until it no longer returns anything. Do the printed timestamps stop at 2018?

---

Export chat template - the format parameters are chat properties. What would the Year-Month-Day be - today's date? I suppose I can add support for them.

[RikuAotsuki]

I can't open a chat lacking earlier messages; the chats themselves are missing. I'll run the import again and let you know how exactly it cuts off, but there's like thirty chats just totally missing from 2017.

As for the date, I should have specified; I'm referring to the chat's creation date.

[suurjaak]

Can you then try the following:

• log in to Skype on the Online-page
• open the Python console
• execute the following:

for id, chat in db.live.skype.chats.recent().items():
    print (id, chat.topic if hasattr(chat, "topic") else chat.user.name)

• execute it over and over again until it no longer returns anything. Does it ever reach the missing chats?

---

Export chat template: you can use created_datetime or first_message_datetime, but the result in filename will have time as well, e.g. 2021-06-30 123355.

[RikuAotsuki]

It does, yes. In fact, I don't know how it decides retrieval order, but the missing chats are mixed in with the ones it did find, like they were skipped instead of the sync just stopping early.

[suurjaak]

The retrieval order is decided by Skype API, ostensibly by recency, but in practice they seem to come in rather random order.

Were there any errors printed in the log window (menu Help -> Show log window) after running online sync?

Can you paste an example from the console of such a missing chat? (You can replace all user IDs with something else, so as not to expose private information here.)

[RikuAotsuki]

No errors in the log, no. As for the console, this particular execution returned mostly chats that didn't sync (group names abbreviated). Also, I'm trying another new online database sync (again) and it might be registering something it didn't before, so I'll update if it just randomly decided to find everything.

for id, chat in db.live.skype.chats.recent().items():
print (id, chat.topic if hasattr(chat, "topic") else chat.user.name)

(u'19:3b8096a51ced4971a3c187cfa2d2f374@thread.skype', u'SB~')
(u'19:dc9241983c0e4ee8a52945003c5e6a8b@thread.skype', u'MFD')
(u'19:06d92f4ef1904102b0d8031989f839e9@thread.skype', u'LF,aS')
(u'19:b11e0bbc75384c97a0f4d6839bac2cc4@thread.skype', u'BB')
(u'19:3ce51f1d134d4b59af965fb52a4c9230@thread.skype', u'SS')
(u'19:4aadf556f2bf424b8bf97819dec82a07@thread.skype', u'AaT')
(u'19:e846d9c3653a46a2917724d331c5a0b5@thread.skype', u'AaT')
(u'19:dc24f73991b842a5959df2451fe37c78@thread.skype', u'TDoKK')
(u'19:0c3c5afda3764255bbd16ed80d26f1f1@thread.skype', u'AL')
(u'19:a6d5451693a349368609b3efb26aac1d@thread.skype', u'NJ')

[suurjaak]

Can you try this:

def getter():
    chatmap = db.live.skype.chats.recent()
    while chatmap:
        for c in chatmap.values():
            yield c
        chatmap = db.live.skype.chats.recent()

chats = getter()

chat = next(chats)

print chat

(You should copy-paste the first block as a whole, and the three last statements one by one)

Repeat the last two statements until you reach a missing chat, then execute the following a few times (first time might not return anything, that is normal):

chat.getMsgs()

Does it return any messages from a missing chat? If yes, can you paste that chat here as it was printed? (It should look like SkypeGroupChat(id=...) or SkypeSingleChat(id=...))

[RikuAotsuki]

I'm guessing I'm doing something wrong, because it seems to just loop?

def getter():
chatmap = db.live.skype.chats.recent()
while chatmap:
for c in chatmap.values():
yield c
chatmap = db.live.skype.chats.recent()
chats = getter()
chat = next(chats)
print chat

chat = next(chats)
Traceback (most recent call last):
File "", line 1, in
StopIteration
print chat
[SkypeGroupChat]
Id: 19:70f4853da85c4b3da5d8305ff7c01dd2@thread.skype
Alerts: True
Topic: Group Thingy
CreatorId: None
UserIds:
AdminIds:
Open: False
History: False
Picture: None
Active: False
chat = next(chats)
Traceback (most recent call last):
File "", line 1, in
StopIteration
print chat
[SkypeGroupChat]
Id: 19:70f4853da85c4b3da5d8305ff7c01dd2@thread.skype
Alerts: True
Topic: Group Thingy
CreatorId: None
UserIds:
AdminIds:
Open: False
History: False
Picture: None
Active: False

[suurjaak]

Github is really messing up the formatting here.. can you attach that last data as a file?

[RikuAotsuki]

Sure. Like this?

console.txt

[suurjaak]

Looks like the copy-paste went badly. First paste this:

def getter():
    chatmap = db.live.skype.chats.recent()
    while chatmap:
        for c in chatmap.values():
            yield c
        chatmap = db.live.skype.chats.recent()

Then this:

chat = next(chats)

Then this:

print repr(chat)

Repeat the last two until you reach a missing chat, then execute

chat.getMsgs()

[RikuAotsuki]

Oh, that one was my bad, I think the first result it returned was a missing chat and I didn't realize it. Here's the result:

console.txt

Edit: Switched from a direct paste to a file, just in case

[suurjaak]

Well, at least with this chat, messages seem to be unavailable via the online API.

What is the result for some other missing chats?

[RikuAotsuki]

What does that mean, exactly?

And as far as I can tell, it's stuck on that one. repeating the second and third commands just brings that chat up again and again.

[suurjaak]

Then something still went sideways in the copy-paste. The steps in detail:

• open a database in Skyperious
• log in to Skype on the Online-page
• open the Python console
• execute

def getter():
    chatmap = db.live.skype.chats.recent()
    while chatmap:
        for c in chatmap.values():
            yield c
        chatmap = db.live.skype.chats.recent()

• then execute

chats = getter()

• then execute these two statements, separately, until you reach a chat that was not synced:

chat = next(chats)

print chat

• once you have reached such a chat, execute this 2-3 times:

chat.getMsgs()

[RikuAotsuki]

No idea how I could be doing it wrong, but here's what it's doing:

console.txt

[suurjaak]

Can you copy-paste the entire console history?

[RikuAotsuki]

I have no idea what's going on now; had to restart my computer the other day so the console wound up cleared, and I can't seem to get it to return ANY information now, with any combination of stuff you've mentioned. Should I just make it create a new database again and then try?

console.txt

[suurjaak]

That is a strange result . Sure, you can try creating a new database from scratch.

[RikuAotsuki]

Alright, did that. Went through the "for id, chat in db.live.skype.chats.recent().items():" bit, then the "def getter():" part. This should be everything from the last returned chats from the former to the looping problem with the latter, though it seems to be a bit different this time around. Everything earlier than that is 18 more groups of returned chats, then the initialization stuff.

console.txt

[suurjaak]

The reason the latter has the looping problem, is because you're doing the "for id, chat.." bit before it. The former consumes all the recent chats, so the latter gets nothing.

Please, do only the steps set down in #100 (comment).

[RikuAotsuki]

Ooooooh. Sorry, wasn't aware that the command needed to be input on a fresh database to function. Will try that and report back.

[RikuAotsuki]

Alrighty, so it's not returning anything, just empty brackets. I tried a second missing chat just in case and got the same result. Here's the first one:

console.txt

[suurjaak]

Well, if that's the case, then nothing to do, those chats are just not syncable from online.

I will keep this issue open until I publish the new program version that fixes the Skype export issue.

[RikuAotsuki]

Huh, I wonder what'd cause that?

Thanks for the help though. Any ETA on the update?

[suurjaak]

Only Microsoft knows what would cause that, and they ain't telling :)

Any ETA on the update?

I guess within a week or two.

[RikuAotsuki]

I guess within a week or two.

Sweet, thanks

[suurjaak]

The newly released Skyperious v4.8 fixes the issue of not being able to create a new database from a Skype export archive.

Can you confirm?

[RikuAotsuki]

Running it now; it's a few hundred thousand messages' worth so it should be done tomorrow xD

[RikuAotsuki]

Yup, looks like it works. Anything "missing" now is pre-database change, I think.

Speaking of which, I have the exported html files of a bunch of chats from 2017. I think I found the database Skype would've extracted them from but can't be totally sure; is there any way to convert them back to a database that can be imported into Skyperious to check?

[suurjaak]

The only way would be to write a parser for the HTML files, which reads in the data and inserts the conversations, contacts and messages to a blank database.

But if you have found the original Skype SQLite database, why not open that in Skyperious?

[RikuAotsuki]

Alrighty. I actually did open in it Skyperious; I was just wondering if there was an easy way to check the old export on the off chance I was wrong about it being the same one. Thanks though!