Add bounds to memory storage #845

jpkrohling · 2018-05-24T16:34:02Z

Resolves #842

Signed-off-by: Juraci Paixão Kröhling juraci@kroehling.de

Which problem is this PR solving?

See Need mechanism to evict spans from in-memory storage based on various conditions #842 for background information

Short description of the changes

This PR adds a simple FIFO eviction mechanism

This is WIP as it's missing:

Documentation changes
Flag to configure the queue size

If the approach from this PR is acceptable, I'll proceed with the doc/flag changes.

jpkrohling · 2018-05-24T16:35:37Z

@objectiser would something like this be sufficient as an initial solution?

black-adder · 2018-05-24T16:39:12Z

I'm ok with these changes since the in memory storage should be used for toying around (no need to optimize for performance), that being said how hard would it be to use a circular buffer instead for the bounded storage?

coveralls · 2018-05-24T17:09:35Z

Coverage remained the same at 100.0% when pulling 67d162e on jpkrohling:842-Add-Bounds-To-Memory-Storage into 037875b on jaegertracing:master.

jpkrohling · 2018-05-24T17:10:35Z

how hard would it be to use a circular buffer instead for the bounded storage

Probably not that hard. Do you have opinions about https://golang.org/pkg/container/ring/ ?

In any case, I would like to do it on a follow-up PR, probably with some heuristics to determine a good default based on the available memory, in case a limit isn't explicitly set. As this is probably causing istio/istio#5782 , a "good enough" solution would be wonderful to have for now :)

objectiser · 2018-05-24T17:53:07Z

@jpkrohling Any solution that prevents the crash will be acceptable at this stage - but would need to be configurable as the amount of memory used by a trace can vary significantly based on number of spans and span size.

objectiser · 2018-05-24T17:59:38Z

plugin/storage/memory/memory.go

@@ -115,6 +130,17 @@ func (m *Store) WriteSpan(span *model.Span) error {
 		m.traces[span.TraceID] = &model.Trace{}
 	}
 	m.traces[span.TraceID].Spans = append(m.traces[span.TraceID].Spans, span)
+	m.ids = append(m.ids, &span.TraceID)


Shouldn't the id only be appended when the trace id first occurs, i.e. in the above conditional block?

Nice catch, will add a test about this

yurishkuro · 2018-05-24T18:04:15Z

+1 to container/ring as the current impl seems to grow the array infinitely
may want to defer memory-bound implementation until Use Protobuf and gRPC for internal communications #773, which will make it much easier to estimate the memory size. But could also build a simple heuristic now that approximates the size of the trace

objectiser · 2018-05-24T18:04:51Z

@jpkrohling Actually using a simple circular buffer may be better - as there could still be a large number of traces being held, so this approach could be very inefficient?

jpkrohling · 2018-05-25T07:57:13Z

Seems like the circular buffer is consensus :) Will do it in this PR.

objectiser · 2018-05-25T09:44:38Z

@jpkrohling FYI istio/istio#5840

Once this PR is merged, if you could sort out a release and let the istio team know - or better still do a PR on the release-0.8 branch to update https://github.com/istio/istio/blob/release-0.8/install/kubernetes/helm/istio/values.yaml#L401.

Might be good if the max number of traces is configurable as part of this PR, and included as a property in the values.yaml file.

jpkrohling · 2018-05-25T10:33:52Z

pkg/memory/config/config.go

+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package config


I'm not sure this object is needed at all, but I'm following the pattern established by the Cassandra configuration object. Consistency wins :)

jpkrohling · 2018-05-25T10:37:16Z

plugin/storage/memory/options.go

+	"github.com/jaegertracing/jaeger/pkg/memory/config"
+)
+
+const limit = "memory.limit"


This translates to a flag like --memory.limit=100. This is also consistent with cassandra, where the plugin name is the prefix for the option. What I don't like, though, is that "limit" is too generic and it's not clear what kind of limit this is. Passing --help will clarify it, but I'm open to suggestions for a better name.

--memory.max-traces perhaps?

Is it traces or spans?

I believe it's better and cleaner to bound on the total number of spans. The docs should also mention an average estimation of span size, so people roughly know what value should be used.

I am not sure how much it would complicate your PR...

Span size estimation is hard when you don't know what people are storing within the span... Also, evicting traces is cleaner than evicting individual spans. Otherwise, a trace might show up now in the UI with 100 spans and then, after a couple of minutes, with only 10 spans.

Span size estimation is hard when you don't know what people are storing within the span...

For example it could use average HTTP span. Which is what most of the people use.

Also, evicting traces is cleaner than evicting individual spans

I am not saying we should evict on per span! Eviction strategy is different than limit :)

Otherwise, a trace might show up now in the UI with 100 spans and then, after a couple of minutes, with only 10 spans.

This can happen with your impl too.

For example it could use average HTTP span. Which is what most of the people use.

I think this fits the memory-bound strategy that was mentioned before. We'll wait for #773 for that.

I am not saying we should evict on per span! Eviction strategy is different than limit :)

OK, so, you mean to keep a count of the number of spans and evict traces when the number of spans reaches the limit? Could be a possible additional strategy and a candidate for a follow-up PR. For now, we just need a simple upper bound.

objectiser · 2018-05-25T11:32:58Z

Maybe inmemory rather than memory to make clear about storage impl and not actual memory?

jpkrohling · 2018-05-25T11:37:54Z

Maybe inmemory rather than memory to make clear about storage impl and not actual memory?

I thought about that, but "memory" is what's used as possible value for SPAN_STORAGE_TYPE :-/

jpkrohling · 2018-05-25T11:41:47Z

I changed this PR to not have a performance impact when the --memory.max-traces options isn't used, so that the ids slice is always empty.

jpkrohling · 2018-05-25T13:39:59Z

Created #846 as a follow-up, to have the circular list to back the in-memory storage.

black-adder · 2018-05-25T16:24:49Z

pkg/memory/config/config.go

+
+package config
+
+// Configuration describes the configuration properties needed to connect to a Cassandra cluster


rm cassandra

black-adder · 2018-05-25T16:27:47Z

plugin/storage/memory/memory.go

+}
+
+// WithConfiguration creates a new in memory storage based on the given configuration
+func WithConfiguration(configuration *config.Configuration) *Store {


I've always preferred passing around configurations as values instead of references. The caller could technically update the config which might lead to errant behavior

And storing it in the parent struct by value as well

yurishkuro

Aside from config by value, lgtm

yurishkuro · 2018-05-25T23:02:41Z

plugin/storage/memory/memory.go

@@ -113,6 +123,22 @@ func (m *Store) WriteSpan(span *model.Span) error {
 	m.services[span.Process.ServiceName] = struct{}{}
 	if _, ok := m.traces[span.TraceID]; !ok {
 		m.traces[span.TraceID] = &model.Trace{}
+
+		// if we have a limit, let's cleanup the oldest traces
+		if m.config.MaxTraces > 0 {


Another reason to pass config by value - NPE

Resolves #842 by adding a simple FIFO eviction mechanism. Signed-off-by: Juraci Paixão Kröhling <juraci@kroehling.de>

Signed-off-by: Juraci Paixão Kröhling <juraci@kroehling.de>

jpkrohling · 2018-05-27T11:18:50Z

@yurishkuro, @black-adder, changes from the last comment implemented. This should now be ready to be merged.

* master: (38 commits) Preparing release 1.5.0 (jaegertracing#847) Add bounds to memory storage (jaegertracing#845) Add metric for debug traces (jaegertracing#796) Change metrics naming scheme (jaegertracing#776) Bump gocql version (jaegertracing#829) Remove ParentSpanID from domain model (jaegertracing#831) Make gas run quiet (jaegertracing#838) Revert "Make gas run quite" Revert "Install gas from install-ci" Install gas from install-ci Make gas run quite Add 'gas' for security problems scanning (jaegertracing#830) Add ability to adjust static sampling probabilities per operation (jaegertracing#827) Support log-level flag on agent (jaegertracing#828) Remove unused function (jaegertracing#822) Add healthcheck to standalone (jaegertracing#784) Do not use KeyValue fields directly and use KeyValues as decorator only (jaegertracing#810) Add ContaAzul to the adopters list (jaegertracing#806) Add ISSUE_TEMPLATE and PULL_REQUEST_TEMPLATE (jaegertracing#805) Upgrade to go 1.10 (jaegertracing#792) ... # Conflicts: # cmd/agent/app/builder.go # cmd/collector/main.go # cmd/query/main.go # cmd/standalone/main.go

jpkrohling requested review from black-adder, pavolloffay, vprithvi and yurishkuro as code owners May 24, 2018 16:34

ghost assigned jpkrohling May 24, 2018

ghost added the review label May 24, 2018

objectiser reviewed May 24, 2018

View reviewed changes

jpkrohling commented May 25, 2018

View reviewed changes

jpkrohling changed the title ~~WIP - Add bounds to memory storage~~ Add bounds to memory storage May 25, 2018

jpkrohling mentioned this pull request May 25, 2018

Add documentation about --memory.max-traces jaegertracing/documentation#81

Merged

jpkrohling mentioned this pull request May 25, 2018

Implement circular list with indexing for the memory storage #846

Closed

yurishkuro approved these changes May 25, 2018

View reviewed changes

ghost assigned black-adder May 25, 2018

black-adder approved these changes May 25, 2018

View reviewed changes

yurishkuro approved these changes May 25, 2018

View reviewed changes

Add bounds to memory storage

36c5fa0

Resolves #842 by adding a simple FIFO eviction mechanism. Signed-off-by: Juraci Paixão Kröhling <juraci@kroehling.de>

jpkrohling added 4 commits May 26, 2018 11:57

Add options object to memory storage

3aeb4ea

Signed-off-by: Juraci Paixão Kröhling <juraci@kroehling.de>

Rename limit to max-traces

2256eed

Signed-off-by: Juraci Paixão Kröhling <juraci@kroehling.de>

Implement simple ring for the in-memory storage

f0e8fca

Signed-off-by: Juraci Paixão Kröhling <juraci@kroehling.de>

Pass configuration as value instead of reference

67d162e

Signed-off-by: Juraci Paixão Kröhling <juraci@kroehling.de>

black-adder merged commit 7796f75 into jaegertracing:master May 27, 2018

ghost removed the review label May 27, 2018

jpkrohling mentioned this pull request May 30, 2018

Log configuration options for memory storage #852

Merged

jpkrohling mentioned this pull request Jun 12, 2018

Sorted key/value store (badger) backed storage plugin #760

Merged

jpkrohling deleted the 842-Add-Bounds-To-Memory-Storage branch July 28, 2021 19:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add bounds to memory storage #845

Add bounds to memory storage #845

jpkrohling commented May 24, 2018 •

edited

Loading

jpkrohling commented May 24, 2018

black-adder commented May 24, 2018

coveralls commented May 24, 2018 •

edited

Loading

jpkrohling commented May 24, 2018

objectiser commented May 24, 2018

objectiser May 24, 2018

jpkrohling May 25, 2018

yurishkuro commented May 24, 2018

objectiser commented May 24, 2018

jpkrohling commented May 25, 2018 •

edited

Loading

objectiser commented May 25, 2018

jpkrohling May 25, 2018

jpkrohling May 25, 2018

pavolloffay May 25, 2018

jpkrohling May 25, 2018

pavolloffay May 25, 2018

jpkrohling May 25, 2018

pavolloffay May 25, 2018

jpkrohling May 25, 2018

objectiser commented May 25, 2018

jpkrohling commented May 25, 2018

jpkrohling commented May 25, 2018

jpkrohling commented May 25, 2018

black-adder May 25, 2018

black-adder May 25, 2018

yurishkuro May 25, 2018

yurishkuro left a comment

yurishkuro May 25, 2018

jpkrohling commented May 27, 2018


		package config

		// Configuration describes the configuration properties needed to connect to a Cassandra cluster

Add bounds to memory storage #845

Add bounds to memory storage #845

Conversation

jpkrohling commented May 24, 2018 • edited Loading

Which problem is this PR solving?

Short description of the changes

jpkrohling commented May 24, 2018

black-adder commented May 24, 2018

coveralls commented May 24, 2018 • edited Loading

jpkrohling commented May 24, 2018

objectiser commented May 24, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yurishkuro commented May 24, 2018

objectiser commented May 24, 2018

jpkrohling commented May 25, 2018 • edited Loading

objectiser commented May 25, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

objectiser commented May 25, 2018

jpkrohling commented May 25, 2018

jpkrohling commented May 25, 2018

jpkrohling commented May 25, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yurishkuro left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jpkrohling commented May 27, 2018

jpkrohling commented May 24, 2018 •

edited

Loading

coveralls commented May 24, 2018 •

edited

Loading

jpkrohling commented May 25, 2018 •

edited

Loading