Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

secp256k1.go appears to be causing some errors for Google App Engine #297

Closed
ThePiachu opened this issue Feb 12, 2015 · 7 comments · Fixed by #298
Closed

secp256k1.go appears to be causing some errors for Google App Engine #297

ThePiachu opened this issue Feb 12, 2015 · 7 comments · Fixed by #298
Labels
build

Comments

@ThePiachu
Copy link

The long []byte array in the file appears to be causing some issues for Google App Engine - https://groups.google.com/forum/#!topic/google-appengine-go/YRsH3cWdql8 .

Is that data necessary for the library to function, or does it simply speed the process up?
@ThePiachu
Copy link
Author

Perhaps the []byte array can be more compressed by switching

https://github.com/btcsuite/btcd/blob/master/btcec/genprecomps.go#L33

to http://golang.org/pkg/compress/zlib/#NewWriterLevel BestCompression

@jrick
Copy link
Member

jrick commented Feb 12, 2015

IIRC internally 6g (and the other gc compilers?) transforms []byte("string") to []byte{'s', 't', 'r', 'i', 'n', 'g'} to avoid the byte slice copy at runtime. However, in this representation 6g can consume a lot of memory since each byte literal is stored as a big integer. We may want to try using a string here instead of a byte slice.

Could you change the byte slice to a string, and change the bytes.Reader to strings.Reader in precompute.go, and report back on if it will compile?

@ThePiachu
Copy link
Author

I couldn't directly change things to string in loadS256BytePoints(), since if _, err := base64.StdEncoding.Decode(decoded, bp); err != nil { requires []byte.

However, I did change secp256k1BytePoints to string and bp := []byte(secp256k1BytePoints) and now everything seems to work so far. Will do some more tests.

@davecgh
Copy link
Member

davecgh commented Feb 12, 2015

I'll work up a pull request real quick which switches this to a string.

@davecgh davecgh added the build label Feb 12, 2015
davecgh added a commit to davecgh/btcd that referenced this issue Feb 12, 2015
@davecgh
btcec: Use a string for the pre-computed table.
c8df48a 
This commit modifies the pre-computed table used to optimize the secp256k1
scalar multiplication to a string instead of a byte slice.  This change
makes the compile more efficient since the Go compiler internally
represents bytes slices inefficiently.

This reduces the memory needed to compile btcec to 3MB versus the previous
40MB before this change.

Fixes btcsuite#297.
@davecgh davecgh mentioned this issue Feb 12, 2015
Merged
@davecgh
Copy link
Member

davecgh commented Feb 12, 2015

@ThePiachu: Can you try with pr #298 please?

davecgh added a commit to davecgh/btcd that referenced this issue Feb 12, 2015
@davecgh
btcec: Use a string for the pre-computed table.
e520bc7 
This commit modifies the pre-computed table used to optimize the secp256k1
scalar multiplication to a string instead of a byte slice.  This change
makes the compile more efficient since the Go compiler internally
represents bytes slices inefficiently.

This reduces the memory needed to compile btcec to 3MB versus the previous
40MB before this change.

Fixes btcsuite#297.
davecgh added a commit to davecgh/btcd that referenced this issue Feb 12, 2015
@davecgh
btcec: Optimize pre-computed table load.
f6a437d 
This commit modifies the pre-computed table used to optimize the secp256k1
scalar multiplication to a string instead of a byte slice.  This change
makes the compile more efficient since the Go compiler internally
represents bytes slices inefficiently.

This reduces the memory needed to compile btcec to 3MB versus the previous
40MB before this change.

In addition, it modifies the code which loads the pre-computed table to
deserialize directly into the table instead of into locals that are then
copied.

Fixes btcsuite#297.
@ThePiachu
Copy link
Author

It looks like it's working. Cool!

@davecgh
Copy link
Member

davecgh commented Feb 12, 2015

Thanks for confirming.

jrick added a commit to jrick/btcd that referenced this issue Aug 9, 2016
@jrick
Start RPC server earlier, add pipes for parent process IPC.
8557cfd 
The RPC server is now the first subsystem that is started.  Because
the RPC server would not be usable without the rest of the subsystems
running, all requests are disabled until bingup has finished.  This
has the added benefit of erroring earlier if the RPC listeners can not
be created.

Rewrite startup/shutdown logic to simplify shutdown signaling.  All
cleanup is now run from deferred functions in the main function.

Add two new config options to set the read and write ends of a pair of
pipes.  This is used as a simple mechanism for a parent process to
communicate with, observe, and manage the lifetime of a child dcrd
process.  When the RX (read end) pipe is closed, clean shutdown
automatically begins.

Add a new flag --lifetimeevents to create and send lifetime event
notifications over the TX (write end) pipe during bringup and
shutdown.  This allows the parent process to observe which subsystems
are currently starting, running, and stopping.

Fixes btcsuite#297.
Fixes btcsuite#298.
jrick added a commit to jrick/btcd that referenced this issue Aug 9, 2016
@jrick
Start RPC server earlier, add pipes for parent process IPC.
7f9c445 
The RPC server is now the first subsystem that is started.  Because
the RPC server would not be usable without the rest of the subsystems
running, all requests are disabled until bringup has finished.  This
has the added benefit of erroring earlier if the RPC listeners can not
be created.

Rewrite startup/shutdown logic to simplify shutdown signaling.  All
cleanup is now run from deferred functions in the main function.

Add two new config options to set the read and write ends of a pair of
pipes.  This is used as a simple mechanism for a parent process to
communicate with, observe, and manage the lifetime of a child dcrd
process.  When the RX (read end) pipe is closed, clean shutdown
automatically begins.

Add a new flag --lifetimeevents to create and send lifetime event
notifications over the TX (write end) pipe during bringup and
shutdown.  This allows the parent process to observe which subsystems
are currently starting, running, and stopping.

Fixes btcsuite#297.
Fixes btcsuite#298.
jrick added a commit to jrick/btcd that referenced this issue Aug 9, 2016
@jrick
Add pipes for parent process IPC.
297a781 
Rewrite startup/shutdown logic to simplify shutdown signaling.  All
cleanup is now run from deferred functions in the main function.

Add two new config options to set the read and write ends of a pair of
pipes.  This is used as a simple mechanism for a parent process to
communicate with, observe, and manage the lifetime of a child dcrd
process.  When the RX (read end) pipe is closed, clean shutdown
automatically begins.

Add a new flag --lifetimeevents to create and send lifetime event
notifications over the TX (write end) pipe during bringup and
shutdown.  This allows the parent process to observe which subsystems
are currently starting, running, and stopping.

Fixes btcsuite#297.
Fixes btcsuite#298.
jrick added a commit to jrick/btcd that referenced this issue Aug 29, 2016
@jrick
Add pipes for parent process IPC.
e0c3df8 
Rewrite startup/shutdown logic to simplify shutdown signaling.  All
cleanup is now run from deferred functions in the main function.

Add two new config options to set the read and write ends of a pair of
pipes.  This is used as a simple mechanism for a parent process to
communicate with, observe, and manage the lifetime of a child dcrd
process.  When the RX (read end) pipe is closed, clean shutdown
automatically begins.

Add a new flag --lifetimeevents to create and send lifetime event
notifications over the TX (write end) pipe during bringup and
shutdown.  This allows the parent process to observe which subsystems
are currently starting, running, and stopping.

Fixes btcsuite#297.
Fixes btcsuite#298.
jrick added a commit to jrick/btcd that referenced this issue Aug 31, 2016
@jrick
Add pipes for parent process IPC.
1defcbb 
Rewrite startup/shutdown logic to simplify shutdown signaling.  All
cleanup is now run from deferred functions in the main function.

Add two new config options to set the read and write ends of a pair of
pipes.  This is used as a simple mechanism for a parent process to
communicate with, observe, and manage the lifetime of a child dcrd
process.  When the RX (read end) pipe is closed, clean shutdown
automatically begins.

Add a new flag --lifetimeevents to create and send lifetime event
notifications over the TX (write end) pipe during bringup and
shutdown.  This allows the parent process to observe which subsystems
are currently starting, running, and stopping.

Fixes btcsuite#297.
Fixes btcsuite#298.
jrick added a commit to jrick/btcd that referenced this issue Aug 31, 2016
@jrick
Add pipes for parent process IPC.
fc9eb1c 
Rewrite startup/shutdown logic to simplify shutdown signaling.  All
cleanup is now run from deferred functions in the main function.

Add two new config options to set the read and write ends of a pair of
pipes.  This is used as a simple mechanism for a parent process to
communicate with, observe, and manage the lifetime of a child dcrd
process.  When the RX (read end) pipe is closed, clean shutdown
automatically begins.

Add a new flag --lifetimeevents to create and send lifetime event
notifications over the TX (write end) pipe during bringup and
shutdown.  This allows the parent process to observe which subsystems
are currently starting, running, and stopping.

Fixes btcsuite#297.
Fixes btcsuite#298.
jcvernaleo pushed a commit to jcvernaleo/btcd that referenced this issue Sep 1, 2016
@jrick
Add pipes for parent process IPC. (btcsuite#332)
0d8104b 
Rewrite startup/shutdown logic to simplify shutdown signaling.  All
cleanup is now run from deferred functions in the main function.

Add two new config options to set the read and write ends of a pair of
pipes.  This is used as a simple mechanism for a parent process to
communicate with, observe, and manage the lifetime of a child dcrd
process.  When the RX (read end) pipe is closed, clean shutdown
automatically begins.

Add a new flag --lifetimeevents to create and send lifetime event
notifications over the TX (write end) pipe during bringup and
shutdown.  This allows the parent process to observe which subsystems
are currently starting, running, and stopping.

Fixes btcsuite#297.
Fixes btcsuite#298.
jcvernaleo pushed a commit to jcvernaleo/btcd that referenced this issue Sep 1, 2016
@jrick
Add pipes for parent process IPC. (btcsuite#311)
6c1aa07 
Rewrite startup/shutdown logic to simplify shutdown signaling.  All
cleanup is now run from deferred functions in the main function.

Add two new config options to set the read and write ends of a pair of
pipes.  This is used as a simple mechanism for a parent process to
communicate with, observe, and manage the lifetime of a child dcrd
process.  When the RX (read end) pipe is closed, clean shutdown
automatically begins.

Add a new flag --lifetimeevents to create and send lifetime event
notifications over the TX (write end) pipe during bringup and
shutdown.  This allows the parent process to observe which subsystems
are currently starting, running, and stopping.

Fixes btcsuite#297.
Fixes btcsuite#298.
sammyne pushed a commit to sammyne/secp256k1 that referenced this issue Sep 14, 2019
btcec: Optimize pre-computed table load.
c015483 
This commit modifies the pre-computed table used to optimize the secp256k1
scalar multiplication to a string instead of a byte slice.  This change
makes the compile more efficient since the Go compiler internally
represents bytes slices inefficiently.

This reduces the memory needed to compile btcec to 3MB versus the previous
40MB before this change.

In addition, it modifies the code which loads the pre-computed table to
deserialize directly into the table instead of into locals that are then
copied.

Fixes [#297](btcsuite/btcd#297)

Source commit at btcd/btcec@f6a437d4c90ba153f45deb6c58c43744e917ed79
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
build
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

btcec: Optimize pre-computed table load. davecgh/btcd
3 participants
@ThePiachu @jrick @davecgh