Batch Download of traces

[chrvo]

Requirement - what kind of business use case are you trying to solve?

Our traces contain data that is covered by GDPR (european user privacy regulations) and, thus developers do not have direct access to all traces. If an error occurs, relevant traces has to be filtered out and the relevant ones are forwarded to the developer.

Another use case:
It would be nice to just attach files with relevant traces to a Jira Ticket. HTTP Links are nice but they are are only available for a certain time, which conflicts with some of our documentation requirements, and everybody has to have access to the jaeger instance.

Problem - what in Jaeger blocks you from solving the requirement?

The process of "exporting" the traces is cumbersome. For each trace, you have to go to the json view and save it.

Proposal - what do you suggest to solve the problem or improve the existing situation?

It would be nice to have:

• A download trace button in the trace view
• A download traces button in the search view to download all found traces -- Batch Download of traces #1274
  • An option to download traces as a zip file with one JSON file per trace in the archive (explained below)
• A download selected traces button in the search view (proposal below)

[Jiali-Xing]

I don't know if there is any updated feature about it. However, since I need to download json files in batch now, I wrote this simple script in Python, which did the work for me:

from bs4 import BeautifulSoup
import urllib3, shutil


# Passing the source code to BeautifulSoup to create a BeautifulSoup object for it.
soup = BeautifulSoup(open("./Jaeger_UI.html"), "html.parser")  # I assume that you save the page source of your Jaeger search result here as `Jaeger_UI.html`. 


# Extracting all the <a> tags into a list.
tags = soup.find_all('a', {'class': 'ResultItemTitle--item ub-flex-auto'})


def download_json(url):
    json_url = 'http://localhost:16686/api/traces/' + url[-16::] + '?prettyPrint=true'
    print(json_url)
    http = urllib3.PoolManager()

    path = url[-16::] + '.json'

    with http.request('GET', json_url, preload_content=False) as r, open(path, 'wb') as out_file:
        shutil.copyfileobj(r, out_file)

# Extracting URLs from the attribute href in the <a> tags.
for tag in tags:
    link = tag.get('href')
    download_json(link)

Hope this helps whoever needs batch-export right now.

[cptkng23]

The option to simply export/download all found traces of a search to json would be so useful.

[Joyce-O]

I would like to work on this issue

[jpkrohling]

It's yours!

[cptkng23]

Hi @Joyce-O

I was wondering if you made some progress with this feature? I can offer to have a look at your approach and/or do a review.

Best, Arne

[cptkng23]

Hi @Joyce-O

Just following up on my above comment. Did you make some progress, can I assist you somehow?

[SapnaGirdhani1]

I am also looking for export/download feature for all found traces in json format. Is there any update on this feature? Thanks!

[Katarzyna-B]

hi @Joyce-O ,
do you make some progress on this topic?
I don't see unfortunately any updates.
I plan to work on it. It is a feature which could be very useful in my company.

[Katarzyna-B]

hi @jpkrohling ,
could be possible to assign this topic to me?
I've already implemented the functionality with download button and it is possible now to download a file with search results in json format.
I hope to hear from you soon.

[yurishkuro]

@Katarzyna-B feel free to submit PRs, there is no real value in assigning issues.

[Katarzyna-B]

@yurishkuro thx for your answer. I will finish writing tests and after that I will create PR.

[yurishkuro]

I am posting here so that people who already subscribe to the issue may provide opinions on the proposed PR #1274.

PR adds a button "Download Results" that saves all traces from the search results as one file. I think there are a couple of issues here:

Selective download

One of the requirements was to select which traces to download. We already have checkboxes next to each trace which when clicked add the traces to a list above the results

I think it would make sense to reuse this for selective downloads, by changing the caption from "N selected for comparison" to just "N selected" and adding a Download button on the right, next to Compare Traces

Download format

I think another motivation for downloading traces is to be able to upload them back in to the UI. This PR downloads them as a single JSON file, which means they can only be uploaded back into UI as a batch (without extra work on splitting the file). Is that what people would want? The alternative could be to download a single zip file where each trace is a separate file.

[Katarzyna-B]

Hi @yurishkuro ,

Thank you for looking into my PR and posting your comments here.

In my PR implementation we have functionality for 'A download traces button in the search view to download all found traces'.
It replaces a workaround from @Jiali-Xing 's script in Python.
Below the picture with a current implementation to get the people possibility to see how it looks with my changes.

What is still missing

• a download trace button in the trace view
• a download selected traces button in the search view

I agree that the other requirements are still needed. I like your suggestion regarding Selective download.

Download format
Could you explain to me more about the uploading file back to the UI. Which case do we cover with this feature? I thought that to find/see traces we should use the 'Find traces' button.’

The alternative could be to download a single zip file where each trace is a separate file.

It will be great if we can make it configurable and a user can decide if he/she prefers one single file in json format or zip file with traces stored separately.

The open question for me is if we want to split the work into small tasks and implement it step by step.
Of course I can implement them too :)
In my current PR we have already MVP and the people can work without a workaround. It will be great if we can merge it.

What do you think about it?

[yurishkuro]

@Katarzyna-B fair enough, we can do this piecemeal, starting with full download. I changed the description to contain multiple tasks.

Could you explain to me more about the uploading file back to the UI.

One of the reasons mentioned for downloading traces in the first place is for storing them in tickets for longer time than the retention period in the main trace database. To view those traces again people would upload them back into the UI (see the tab in the Search form). And this is where it could be helpful if the traces were stored as individual files in an archive (although upload should work for a JSON array too). It's more of a nice-to-have functionality (I made a separate subtask for it), could be done by adding a dropdown option to the download button, e.g. like this:

[cptkng]

Hi @yurishkuro and @Katarzyna-B

I commented above as one of the last commentators that the download of search results would be really helpful.

So I would be happy if the PR from @Katarzyna-B would get merged. That would be extremely helpful already. And maybe a good start for the other two aspects to be worked on.

[Katarzyna-B]

Hi @yurishkuro
Sorry for a late response. I was few days off.

@Katarzyna-B fair enough, we can do this piecemeal, starting with full download. I changed the description to contain multiple tasks.

Thank you. I will check the PR-comments to get the first task already done.
How often do we have releases?

Could you explain to me more about the uploading file back to the UI.

One of the reasons mentioned for downloading traces in the first place is for storing them in tickets for longer time than the retention period in the main trace database.
...

I got your point. I think it is really good idea.

could be done by adding a dropdown option to the download button.

It is good place in case you've already searched for some traces if not you see only the logo on the right side.
In this case we can place the upload/import button near to Find Traces button.
Maybe later we can discuss about the place(s) of the upload/import button to find an optimal solution.

[Katarzyna-B]

Hi @yurishkuro,
thank you for merging my PR.
Can you tell me when a new version is planned?

I don't see any task regarding the upload/import functionality.
It could be a good addition to the current download button. Should we do it in scope of this issue?

In my previous comment I suggested different place for the upload/import button.

could be done by adding a drop-down option to the download button.

It is good place in case you've already searched for some traces if not you see only the logo on the right side.
In this case we can place the upload/import button near to Find Traces button.
Maybe later we can discuss about the place(s) of the upload/import button to find an optimal solution.

what do you think?

[yurishkuro]

I don't follow. The upload is at the top-left, next to Search. I do think it should be renamed from "JSON File" to "Upload" (maybe also add icons to both Search and Upload)

[Katarzyna-B]

Hi @yurishkuro ,
I got your point. I thought we need to add the new functionality for uploading, but we have it already in UI ("JSON File").
Sorry for the confusing
I checked if the format of a downloaded file is correct for uploading it back to UI.
I needed to change this file a little bit and now we can download and upload it back.
You can find the fix in my new PR.
Have a nice day.