Crypto.Util.Counter.new returns a dictionary instead of an object (how to keep track of/get IV value for AES-CTR?)

[Source61]

Python version: 3.9.2
Pycryptodome version: 3.15.0
Arrived here migrating from pycrypto which has recently stopped working (syntax error in the code) unless I used to run python2 for my program, I'm not entirely sure.
Issue: Crypto.Util.Counter.new(...) returns a dictionary when the documentation clearly says "Returns: An object that can be passed with the :data:counter parameter to a CTR mode cipher."

[Source61]

I checked older versions and it seems pycryptodome has been using a dict since the beginning.
This differs from pycrypto which had Crypto.Util.Counter.new(...) return an actual (class instance) object with methods like next_value().
My program relies on this to be able to keep track of the IV value since the same IV is supposed to not be used more than once.
How do I keep track of the IV value without this to ensure the same IV is never used twice in a release? (I use AES in Python to encrypt client-based game asset files which are decrypted in a CPP-based game client using WBC AES-CTR, the IV also needs to be written down inside a file for the game to be able to decrypt the files)

The documentation for Crypto.Ciphers.AES says regarding the iv parameter:
"If not provided, a random byte string is generated (you must then read its value with the :attr:iv attribute)."
However the object returned from Crypto.Ciphers.AES.new doesn't have an iv attribute or any other method or attribute I could find related to the iv/nonce (there is a nonce attribute, but its value is always b'').

[Legrandin]

However the object returned from Crypto.Ciphers.AES.new doesn't have an iv attribute or any other method or attribute I could find related to the iv/nonce (there is a nonce attribute, but its value is always b'').

For CTR mode, the name of the attribute is nonce.

See the examples here: https://pycryptodome.readthedocs.io/en/latest/src/cipher/classic.html#ctr-mode

[Source61]

Hey, sorry to re-iterate, but my question wasn't literal - the problem is the nonce attribute is ALWAYS empty.
I already wrote this in the issue, although admittedly I know my issue was a little messy, as I always try to do as little research as possible until necessary, so as I did more research I started explaining more things.

Not to complain, but considering the long response time I decided to write my own Python wrapper for AES instead to solve this issue. Wrote it within a couple of hours, should have taken much less, about 5 minutes actually, but I overcomplicated things and ended up spending a couple of hours.

So the issue in your software isn't solved, but the issue is solved for me as I decided to write my own software that actually works like it should instead.

Again sorry for being cheeky, I don't appreciate it when people think I'm wasting their time by not reading anything I said properly and being dismissive off hand.

Good luck with your software.

[Legrandin]

If I do:

from Crypto.Cipher import AES
from Crypto.Random import get_random_bytes
cipher = AES.new(key, AES.MODE_CTR)
print(cipher.nonce)

I get:

b'\xbdu\xcd1g\xa2\xe8f'

Which is clearly not empty. Next time, if you have an issue, the best way to receive help is to provide an example demonstrating the issue and how you got to it. But I am glad you sorted out the problem in the end.

[Source61]

Which is clearly not empty. Next time, if you have an issue, the best way to receive help is to provide an example demonstrating the issue and how you got to it. But I am glad you sorted out the problem in the end.

Sorry that you missed it, but I clearly did in my second comment, stating that the nonce attribute is empty.
In my first comment/main issue I was asking an open-ended question, I hadn't yet decided it was a bug, so I asked a question followed up by the actual bug report inside another question - I try not to make assumptions, I don't know if you know the idiom that assumptions makes an ass out of me and you, I think this is a pretty good demonstration of that.

As far as the actual bug report now, setting aside who said what first which ended up being more important than the issue in the end since you completely missed what I said the first time (not just the spirit but literally what I said, that the nonce attribute always showed an empty bytes value, feel free to read back if you still haven't caught it)... I cannot replicate the bug at the moment using your code. I definitely never used this code snipped though, I used the code snippet from here https://pycryptodome.readthedocs.io/en/latest/src/util/util.html#crypto-util-counter-module
And the cipher object in this case have no nonce attribute at all.

I can't replicate having an empty nonce attribute -- actually let me re-iterate one more time just before you miss the crucial part right above here first: the nonce attribute is completely missing in the code snippet I provided. But, technically I can't replicate the empty nonce attribute with this code, since it's not even there.
I run about 5 different machines/OS and I instantly reverted my code that was intermediately using pycryptodome code once I recognized that the nonce attribute was empty and I couldn't use the code, so I don't have the code that produced this bug available anymore, and I'm not sure which machine/OS I installed pycryptodome on and which not etc.
It's possible something was broken with my package manager/pip, maybe I still had some partial or full package of the original pycrypto left or something, I don't know, but I'm not gonna bother investigating.
I've made my point, you can take it or leave it with as much snark as you wish, clearly having no introspective ability to recognize you were in the wrong for going on the offense here in the first place, no matter how "confusing" or "lacking" you found my issue for daring to make an issue more as an original question than a bug report, how dare I, even though it clearly said in my second comment what the problem was, and the way to replicate seemed trivial because all I had used was copy-pasted code from your own documentation, which in this case completely lacks the nonce attribute.

Good luck, hope not to encounter someone like you again with such relentless conviction in yourself and arrogance, namaste.

[Source61]

Also in case you still managed to miss the snippet and would like to blame me again for your own poor ability to read what's right in front of your nose, here's the code I used to notice there's no nonce attribute in cipher in this code snippet:

from Crypto.Util import Counter
from Crypto import Random

nonce = Random.get_random_bytes(4)
ctr = Counter.new(64, prefix=nonce, suffix=b'ABCD', little_endian=True, initial_value=10)
key = b'AES-128 symm key'
plaintext = b'X'*1000000
cipher = AES.new(key, AES.MODE_CTR, counter=ctr)
ciphertext = cipher.encrypt(plaintext)

print(dir(cipher))

You can substitute the last line with cipher.nonce if you wish and you'll get an object has no attribute error.
The value of the nonce line (get_random_bytes) doesn't change after encryption, so you couldn't get the nonce value that anyway.

Anyway feel free to spare me any explanation, I obviously don't care anymore, we're both just engaging in a pissing contest at this point since you started off with all the aggression, can't seem to fathom that you could be in the wrong, and probably will even fail to recognize you're participating in a pissing contest right now, speaking volumes about just how deep you're in it, but anyway, take care, hf.

[Akuli]

I spent a while looking at this. It has to do with whether counter is passed or not.

>>> from Crypto.Util import Counter
>>> from Crypto.Cipher import AES
>>> AES.new(b'1'*32, AES.MODE_CTR, counter=Counter.new(128)).nonce
b''
>>> AES.new(b'1'*32, AES.MODE_CTR).nonce
b'\xfb0K\xa4\x9c>\xdd\x17'

What is unclear to me is why the blank nonce attribute is a problem. On this line:

ctr = Counter.new(64, prefix=nonce, suffix=b'ABCD', little_endian=True, initial_value=10)

we already know the nonce when we are creating the counter, so why not simply pass it around to whatever else needs it?

It might also be silly for the counter's nonce to be available through the cipher. Here the nonce is passed to the counter as a prefix, but the counter also takes suffix and initial_value parameters. How should the cipher know which one of these to use as the nonce? As far as I can guess, it's up to the user to decide which Counter parameter the nonce goes to.

[Source61]

I spent a while looking at this. It has to do with whether counter is passed or not.

>>> from Crypto.Util import Counter
>>> from Crypto.Cipher import AES
>>> AES.new(b'1'*32, AES.MODE_CTR, counter=Counter.new(128)).nonce
b''
>>> AES.new(b'1'*32, AES.MODE_CTR).nonce
b'\xfb0K\xa4\x9c>\xdd\x17'

Thanks for replicating.

What is unclear to me is why the blank nonce attribute is a problem. On this line:

ctr = Counter.new(64, prefix=nonce, suffix=b'ABCD', little_endian=True, initial_value=10)

we already know the nonce when we are creating the counter, so why not simply pass it around to whatever else needs it?

It might also be silly for the counter's nonce to be available through the cipher. Here the nonce is passed to the counter as a prefix, but the counter also takes suffix and initial_value parameters. How should the cipher know which one of these to use as the nonce? As far as I can guess, it's up to the user to decide which Counter parameter the nonce goes to.

Well, basically convenience, it's easier than having to implement your own class that takes in the unencrypted data, checks the length, and calculates how much to increment the nonce with, honestly it's at least 50% of the whole implementation (at least in the case of my implementation which was 2 line of CY, about 20 lines of PY and about 7 lines of C++ - didn't take much code at all, and that includes all necessary asserts to make sure the datatype is right), so feels kind of a waste to have to implement the whole nonce functionality on your own, not to mention I honestly think it's a little weird of a question to ask in the first place - why is it a problem that your library doesn't have any functionality to track the nonce in CTR mode? Really?
Seems a bit of a strange question to me, although you're right, it's not strictly necessary since it CAN technically be implemented by the user.
But at that point I'd rather just make my own little library with half the effort and knowing I can rely on this library for the rest of my life without having to bother anyone in the future that doesn't want to hear about any potential bugs (referring to @Legrandin here).

Anyway, thanks for replicating, (sincerely) best of luck.

Also, just a tip: another option is to try to keep backwards compatibility with pycrypto whenever possible.
That's the option I would pursue, but everyone's different.

[Akuli]

why is it a problem that your library doesn't have any functionality to track the nonce in CTR mode?

As far as I understand it, pycryptodome does track it internally, because otherwise the encrypt() method would return the same thing every time:

>>> from Crypto.Util import Counter
>>> from Crypto.Cipher import AES
>>> AES.new(b'1'*32, AES.MODE_CTR, counter=Counter.new(128))
<Crypto.Cipher._mode_ctr.CtrMode object at 0x7f80b61c5f10>
>>> cipher = AES.new(b'1'*32, AES.MODE_CTR, counter=Counter.new(128))
>>> cipher.encrypt(b'asd')
b'o\xad\x16'
>>> cipher.encrypt(b'asd')
b'{+|'
>>> cipher.encrypt(b'asd')
b'\x9f\xf2I'

My question is why do you as the user need to access the internal nonce. What do you do with the changed/incremented nonce? If you encrypt more data with it, why not just reuse the cipher object?

[Source61]

why is it a problem that your library doesn't have any functionality to track the nonce in CTR mode?

As far as I understand it, pycryptodome does track it internally, because otherwise the encrypt() method would return the same thing every time:

Sure, but the nonce isn't available so it's worthless. It doesn't do/achieve anything for us except for internal state.
In fact that's an even better argument for having the nonce available. First it's already internally tracked, and then you have to re-implement the internal tracking all on your own without pointers or any functionality within the library to aid in this process and cross your fingers and hope your implementation works just like the internal tracking, since it has to match.
It just doesn't make sense to leave the user up to implement this like this. Even libraries that allows you to create your own counter methods has their own default counting method so you don't have to do it yourself, it's entirely optional.

>>> from Crypto.Util import Counter
>>> from Crypto.Cipher import AES
>>> AES.new(b'1'*32, AES.MODE_CTR, counter=Counter.new(128))
<Crypto.Cipher._mode_ctr.CtrMode object at 0x7f80b61c5f10>
>>> cipher = AES.new(b'1'*32, AES.MODE_CTR, counter=Counter.new(128))
>>> cipher.encrypt(b'asd')
b'o\xad\x16'
>>> cipher.encrypt(b'asd')
b'{+|'
>>> cipher.encrypt(b'asd')
b'\x9f\xf2I'

My question is why do you as the user need to access the internal nonce. What do you do with the changed/incremented nonce? If you encrypt more data with it, why not just reuse the cipher object?

You need it in CTR to keep track of what nonce is used up and should not be re-used between releases.
So for each release the nonce must continuously be incremented.
Part of the requirement for CTR is that you must NEVER re-use the same nonce, otherwise the encryption is no longer secure and can be attacked. So if you can't keep track of the nonce a different mode should be picked.
I picked CTR because that suits my use case, I just have to keep incrementing the nonce between releases, and CTR supports multi-threaded/-core encryption for higher performance unlike most of the other modes.
I use it to encrypt/decrypt game assets using WBC AES. The decryption time is already ca. 2-3 seconds single core on a ultra-modern laptop (Ryzen 7 4700U CPU), so it would be nice to reduce that loading/decryption time by using multiple cores in the future, especially for players with lower spec computers, assuming the WBC supports it, either way it made sense at the time looking at the specs, it's also just nice to be familiar with CTR for multicore performance in potentially other projects as well.
But just to re-iterate, to use CTR the nonce must never be re-used, that's why we need access to it, and why every other AES CTR library for python lets you access the nonce.

[Akuli]

I am well aware that you need a different nonce every time :)

So for each release the nonce must continuously be incremented.

What is the difference between doing this and just constructing one cipher object that is used for all encrypting/decrypting? To me, continuous incrementing sounds like exactly what you would get if you just keep reusing the same cipher.

Is it because you are encrypting with pycryptodome and you need to send the current nonce value in the beginning of each chunk of encrypted data? If that is the case, why not use random nonces?

[Source61]

What is the difference between doing this and just constructing one cipher object that is used for all encrypting/decrypting? To me, continuous incrementing sounds like exactly what you would get if you just keep reusing the same cipher.

Is it because you are encrypting with pycryptodome and you need to send the current nonce value in the beginning of each chunk of encrypted data? If that is the case, why not use random nonces?

What do you mean, pickle the cipher object or something?
The problem is I need to actually know the nonce on the client side, otherwise the decryption won't work.
Which is how it's supposed to work, the nonce shouldn't be kept secret but shared with the client for successful decryption (as IV) (yes, what you asked in the second paragraph).
Theoretically you could decrypt it using only python by pickling the object (but omg honestly what a solution, having to send pickled objects to the client), but my game client software is written in C++ and doesn't support Python (no I'm not gonna add Python scripting support to it so that I can pickle a cipher object instead of what actually makes sense which is for the vendor to give users access to the nonce - for God's sake even PyCrypto had this functionality and then it was somehow removed in PyCryptodome when a counter is passed, it just doesn't make sense).

Anyway as you can see the maintainer of this repo isn't interested in fixing his software and I'm not interested in trying to convince him to, it's his loss if he doesn't want his software to work properly and can't own up and apologize for having a stinking attitude in the first place when a bug report is made.
I don't really feel like continuing this conversation for that reason.
You've been reasonable, but the maintainer isn't, so I'm not interested in helping him in any way, and nobody else should be either imo, he shows very poor leadership skills/attitude with arrogance and immediate dismissiveness, people like that doesn't interest me.

But to answer your last question, random nonces are less secure. I don't know by how much, I don't want to have to do the math involved, but the more releases the higher the chance for collision and significantly so (birthday problem).

I like that you think outside the box, but I think outside the box thoroughly as well and I don't think there's any solution here without having access to the nonce.
But fortunately it's not a problem for me since I wrote my own library in a couple of hours that seems stable and works like it should and gives access to the nonce as an int, so I'm not in need for any solutions here, I already have my solution.

[Akuli]

I didn't mean that the cipher object should be pickled, and I agree that would be ridiculous :)

This is how I imagine it:

1. Client and server decide to use nonce X. They both create a cipher initialized with nonce X.
2. Client sends data A to server. The client encrypts with nonce X, which changes the cipher's state to X+len(A). The server decrypts with nonce X, so its internal state also changes to X+len(A).
3. Server sends data B to client. Both ciphers are already at state X+len(A), so that is the nonce they use. Both ciphers end up in state X+len(A)+len(B).

I think the problem is that you want to send the nonce together with the data. The server is supposed to say "let's use nonce X+len(A)" when it sends data B, but there is no way to ask pycryptodome what X+len(A) is.

This makes me wonder: do you really need to send the nonce in the beginning of every message if it increments predictably? Because of compatibility with other software that uses the same protocol?

Also, thanks for taking the time to ELI5 this to me :)

[Varbin]

@Akuli @Source61

Somehow this sounds like an XY problem.

Is this strict nonce-keeping really an issue with a sufficiently sized random nonce? Consider the following code:

from Cryptodome.Cipher import AES

from os import urandom


NONCE_LENGTH = 12


def encrypt(key: bytes, data: bytes) -> bytes:
    nonce = urandom(NONCE_LENGTH)
    return nonce + AES.new(key, mode=AES.MODE_CTR, nonce=nonce).encrypt(data)


def decrypt(key: bytes, cipher: bytes) -> bytes:
    nonce = cipher[:NONCE_LENGTH]
    return AES.new(key, mode=AES.MODE_CTR, nonce=nonce).decrypt(cipher[NONCE_LENGTH:])


if __name__ == "__main__":
    data = b'123'
    key = b'\x81'*16

    encrypted = encrypt(key, data)
    decrypted = decrypt(key, encrypted)

    assert decrypted == data

96-bit random nonces are used here. This allows up to $2^{32}$ calls to encrypt with the same key with the probability of until the probability of a nonce reuse reaches $2^{-32}$. Each call to encrypt can include 64 GiB of data.
If you need to ensure nonce reuse will never ever occur, you can still keep track of the first 12 bytes of the call to encrypt and never call it with more then 64GiB of data.

[Source61]

I didn't mean that the cipher object should be pickled, and I agree that would be ridiculous :)

This is how I imagine it:

1. Client and server decide to use nonce X. They both create a cipher initialized with nonce X.

2. Client sends data A to server. The client encrypts with nonce X, which changes the cipher's state to X+len(A). The server decrypts with nonce X, so its internal state also changes to X+len(A).

3. Server sends data B to client. Both ciphers are already at state X+len(A), so that is the nonce they use. Both ciphers end up in state X+len(A)+len(B).

I think the problem is that you want to send the nonce together with the data. The server is supposed to say "let's use nonce X+len(A)" when it sends data B, but there is no way to ask pycryptodome what X+len(A) is.

This makes me wonder: do you really need to send the nonce in the beginning of every message if it increments predictably? Because of compatibility with other software that uses the same protocol?

Theoretically no I suppose, but I'm not going to rewrite my code in C++ for this.
This is still a very strange way of saying that pycryptodome doesn't need to provide a nonce, afaik it's still standard to provide a nonce for every encryption/decryption, so it makes sense that people would follow this standard, as did I.
Edit: But while only retaining to your question, I would still need to be able to determine what the nonce is at the start and the end to continue the sequence, I would know the start, but not the end.

Also, thanks for taking the time to ELI5 this to me :)

Np, I'm 5 myself so I like when people ELI5 I suppose. There's a lot of smart people around, but there's also a lot of less smart people around, I might be one of them depending on where you set the bar, but it's often not easy to know exactly how smart or knowledgeable someone is in such a brief conversation so I like to default to ELI5 I suppose if that makes sense.

@Akuli @Source61

Somehow this sounds like an XY problem.

Is this strict nonce-keeping really an issue with a sufficiently sized random nonce? Consider the following code:

from Cryptodome.Cipher import AES

from os import urandom


NONCE_LENGTH = 12


def encrypt(key: bytes, data: bytes) -> bytes:
    nonce = urandom(NONCE_LENGTH)
    return nonce + AES.new(key, mode=AES.MODE_CTR, nonce=nonce).encrypt(data)


def decrypt(key: bytes, cipher: bytes) -> bytes:
    nonce = cipher[:NONCE_LENGTH]
    return AES.new(key, mode=AES.MODE_CTR, nonce=nonce).decrypt(cipher[NONCE_LENGTH:])


if __name__ == "__main__":
    data = b'123'
    key = b'\x81'*16

    encrypted = encrypt(key, data)
    decrypted = decrypt(key, encrypted)

    assert decrypted == data

96-bit random nonces are used here. This allows up to 232 calls to encrypt with the same key with the probability of until the probability of a nonce reuse reaches 2−32. Each call to encrypt can include 64 GiB of data. If you need to ensure nonce reuse will never ever occur, you can still keep track of the first 12 bytes of the call to encrypt and never call it with more then 64GiB of data.

I would like to know what math you used to come to this conclusion. I'm guessing you shifted 4 bytes off the 16 bytes and determined that's the probability for a collision now, I'm not sure, I don't even remember what the purpose of this slicing of the nonce was in theory to be honest so I'd need a reminder, I just use the entire 16 bytes.
Is there even a point of slicing it if both sides are/the entire thing is random generated?
Unless there's something specific happening with the slicing that solves this I doubt the probability of a collision would be 2-^32 due to the birthday paradox as mentioned earlier.
In this case I'm the one that needs to have it ELI5 if there's something I'm missing.

Edit: Never mind, you're right about the probability, or supposedly it's 2^-48, even worse for my case.
I just defaulted to a simpler solution using the entire 16 byte nonce as a counter so I didn't have to worry about the math :-)
So I suppose you're right we can use random nonces unless generating a massive amount of data each time split into many segments or parts, which I'm not (only a few hundred files).
But it's still a convoluted way of saying we don't want our software to work optimally, for example in the case that someone generates 2^32 number of encrypted files for each round, in this case the user would be left out of options but use a different library as well.
And that's not the worst part, the worst part is just the whole attitude of the maintainer which doesn't belong in an open source project at all imo, even if he was in the right, which he isn't - this implementation is still suboptimal - but even if he was, he didn't come about it the right way.
Technical accuracy is important, but to me it's at least as important people aren't being a single-minded twat only seeing things exclusively their way when discussing the accuracy of everything technical, which the maintainer didn't even do (discuss anything technical, he just defaulted to his own lacking documentation that doesn't mention this bizarre behavior, which differs from legacy pycrypto behavior, that the nonce is no longer accessible when supplied with a nonce, closed the issue and left off), only the two of you actually discussed the technicalities relevant to the issue.

Edit2: And just to be clear, this is still clearly a bug. Why have an attribute called 'nonce' that always returns b''?
Why call the class Crypto.Util.Counter if it doesn't actually count but just sets IV? Why have this class at all?
I'm almost certain the only reason it exists is because of legacy PyCrypto, which is what this library was based on, which is what I used and why I arrived here because PyCryptoDome was supposed to just be a continuation of PyCrypto where PyCrypto is no longer maintained, with PyCrypto Crypto.Util.Counter was actually working like it should, was a class object and not a dict, and had a 'nonce' attribute that never returned b'', certainly not because a Counter object was passed to it.
So you can keep changing the focus towards me, that's fine, I enjoy having my mind challenged - you're right I could have used a random nonce, just make sure you don't convince yourself that because of that this software works like it should just because it's possible to make most implementations work with it, its behavior still doesn't make any sense, it's not really backwards compatible with PyCrypto when it easily could and by any reasonable standard should be when it should be an easy fix, I literally replaced the entire AES-CTR module in about 60 lines of code (excl. the AES library in C which I just #included), like none of what's going on in this issue makes much of any sense, I kind of appreciate the chat, and I don't mean to be rude, but just as a matter of the original issue the maintainer was clearly in the wrong here by all standards of logic and etiquette, except for the one caveat that a maintainer can technically want and do anything he likes within the limit of the law, but as far as who or what we support here I really don't understand this discussion, I'm going to assume you just wanted to help or something, which is cool, but if you're just trying to defend the maintainer then I don't understand why we're only focusing on my part in this and what I technically theoretically potentially could have done instead, instead of focusing on the actual original topic of this issue, which is bad documentation, buggy behavior (b'' return value, not documented, why?), and needlessly poor backwards compatibility with pycrypto - that's the original topic guys.