Conversation data export not working via UI #39

urakagi · 2018-10-29T06:12:24Z

Hi, I give up to make polisReport working and want to export data from my local polis instance.
But it's a pity that the export function is not working either. It shows no any error but I just don't have the mail.
Is there any hint or SMTP setup I need to do? Thanks.

To Do

^{todos added by maintainers}

temporarily remove data export from client-admin UI
add mention to export page of knowledgebase
document manual data export via command line (server administrator)
fix data export UI

The text was updated successfully, but these errors were encountered:

urakagi · 2018-10-30T08:42:15Z

I'm now generating csv file from database directly. Can I please have a hint that how I get group-id for participants-votes.csv? Thank you very much.

metasoarous · 2018-10-30T22:27:55Z

Hi @urakagi. We're aware that there's a issue with the export functionality right now, and I'm working on that. Sorry for the hassle.

I just pushed some changes to master that make it possible to export from the command line like so lein run export -f thefilename.zip -Z 2convurlid. You'll have to have all the same environment variables; Also note that this only exports from the prod math results buckets, so make sure you've been running the math worker and server with MATH_ENV=prod. If you need to export from the preprod or dev (or whatever other math env buckets), let me know and I'll add a flag for this.

urakagi · 2018-10-31T01:38:15Z

Thank you @metasoarous, I tried to run this and get this error:

18-10-31 01:35:49 chttl-b6a0edd74707c886.local.com INFO [polismath.runner:158] - Runner main function executed
18-10-31 01:35:49 chttl-b6a0edd74707c886.local.com INFO [polismath.runner:174] - Running subcommand: export
Exception in thread "main" java.lang.NullPointerException, compiling:(/tmp/form-init7699330447764564917.clj:1:73)
        at clojure.lang.Compiler.load(Compiler.java:7526)
        at clojure.lang.Compiler.loadFile(Compiler.java:7452)
        at clojure.main$load_script.invokeStatic(main.clj:278)
        at clojure.main$init_opt.invokeStatic(main.clj:280)
        at clojure.main$init_opt.invoke(main.clj:280)
        at clojure.main$initialize.invokeStatic(main.clj:311)
        at clojure.main$null_opt.invokeStatic(main.clj:345)
        at clojure.main$null_opt.invoke(main.clj:342)
        at clojure.main$main.invokeStatic(main.clj:424)
        at clojure.main$main.doInvoke(main.clj:387)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at clojure.lang.Var.applyTo(Var.java:702)
        at clojure.main.main(main.java:37)
Caused by: java.lang.NullPointerException
        at polismath.system$create_and_run_system_BANG_.invokeStatic(system.clj:63)
        at polismath.system$create_and_run_system_BANG_.invoke(system.clj:61)
        at polismath.runner$_main.invokeStatic(runner.clj:175)
        at polismath.runner$_main.doInvoke(runner.clj:156)
        at clojure.lang.RestFn.invoke(RestFn.java:482)
        at clojure.lang.Var.invoke(Var.java:403)
        at user$eval61568.invokeStatic(form-init7699330447764564917.clj:1)
        at user$eval61568.invoke(form-init7699330447764564917.clj:1)
        at clojure.lang.Compiler.eval(Compiler.java:7062)
        at clojure.lang.Compiler.eval(Compiler.java:7052)
        at clojure.lang.Compiler.load(Compiler.java:7514)
        ... 12 more

Then lein stops. I copied run script with the enviroment variables. MATH_ENV=prod. The main polisMath instance is running. What can be wrong?

urakagi · 2018-10-31T01:42:38Z

Also, what is the repo you pushed? I don't see any new commit on polisServer or polisMath.
I have completed to rewrite an exporting function other than group_id, so I'll be happy to have an explanation about how to get it.

metasoarous · 2018-10-31T06:04:02Z

My bad, I pushed to the wrong repo earlier. It's up there now if you pull down master again.

urakagi · 2018-10-31T07:14:44Z

@metasoarous It worked, thank you very much.
Also I have sent a PR pol-is/polisMath#3, to write down this command as script file and clearify what environment variables need to be set.

metasoarous · 2018-10-31T17:14:53Z

I'm going to reopen this issue, as the underlying issue with polisServer isn't resolved yet.

dockerfile

patcon · 2020-05-06T04:57:26Z

Is data export issue still unresolved @metasoarous?

patcon · 2020-05-06T21:50:49Z

Colin confirmed in https://github.com/pol-is/polis-issues/issues/137#issuecomment-624907223 that it's not working at the moment.

Any pointers on where the issue might be?

patcon · 2020-08-15T17:50:41Z

Came up in chat when a newcomer was confused by mention of export in knowledgebase.

Gitter chat:

@arjentum
Also, I'm not sure if I'm missing something really obvious, but how does one export the data from a Pol.is report? I think I've looked through all of the conversation admin pages already...

@patcon
No no you're not crazy! @colinmegill rewrote client-admin lately, and we removed the export interface until someone has time to fix a bug that was preventing users from exporting: #39 (comment)

@patcon
Also, we should probably add a prominent mention of this issue to the knowledge base.

We should add a mention of this in knowledgebase export page

patcon · 2020-09-18T23:29:30Z

More gitter chat

we're happy to provide you with exports for this project till we get that system fixed up, which we're aiming to do within the next few months.

patcon · 2020-09-18T23:39:28Z

https://gitter.im/pol-is/polisDeployment?at=5f651c0ef969413294f45771

Yes; To confirm what @joshsmith2 said, it is possible for us to get xids out in the exports, it's just not allowed in the UI currently.

To be clear, we don't automatically include xid in the exports, since they can be potentially sensitive (emails, etc).

I think we'll probably at least have an option for doing this though, and using personal information like emails directly for xid is probably a bad pattern anyway when it can be avoided.

colinmegill · 2020-09-19T07:46:58Z

Maybe we should break the data export up. Only some of the files are too big to send over the wire. We could have the summary, and comments, be more visible / explorable in browser as raw data and copy-able as CSV. The raw matrix could be too, up to a certain size...

Bowling Green, 2000 people and 1000 comments, is 2 mb

colinmegill · 2020-09-19T07:48:22Z

raw.pol.is/convoID_votes.csv?

colinmegill · 2020-09-19T07:49:05Z

Or pol.is/raw/convoID/votes.csv?

patcon · 2020-12-12T16:52:52Z

+1 from @hamnox in Gitter chat 🎉 thx!

patcon · 2021-05-15T04:11:01Z

For future travellers looking for information on data export, I've created an explanation here in a community-maintained FAQ: https://hackmd.io/@patcon/conversa-calls/https%3A%2F%2Fhackmd.io%2F%40patcon%2FB1rwrXiu_#How-do-I-export-my-data ❤️

dockerfile

urakagi closed this as completed Oct 31, 2018

metasoarous reopened this Oct 31, 2018

patcon pushed a commit to pol-is-trial-balloon/polis that referenced this issue Apr 9, 2020

Merge pull request compdemocracy#39 from rohanrichards/master

73833fe

dockerfile

patcon changed the title ~~Export CSV not working in local server~~ Conversation data export not working via UI Aug 15, 2020

raulsperoni pushed a commit to proyectourgente/polis that referenced this issue Sep 1, 2021

Merge pull request compdemocracy#39 from rohanrichards/master

765bc81

dockerfile

metasoarous added this to Priority in Polis Development Dec 9, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Conversation data export not working via UI #39

Conversation data export not working via UI #39

urakagi commented Oct 29, 2018 •

edited by patcon

Loading

urakagi commented Oct 30, 2018

metasoarous commented Oct 30, 2018

urakagi commented Oct 31, 2018

urakagi commented Oct 31, 2018 •

edited

Loading

metasoarous commented Oct 31, 2018

urakagi commented Oct 31, 2018

metasoarous commented Oct 31, 2018

patcon commented May 6, 2020 •

edited

Loading

patcon commented May 6, 2020

patcon commented Aug 15, 2020 •

edited by metasoarous

Loading

patcon commented Sep 18, 2020 •

edited by metasoarous

Loading

patcon commented Sep 18, 2020

colinmegill commented Sep 19, 2020

colinmegill commented Sep 19, 2020

colinmegill commented Sep 19, 2020

patcon commented Dec 12, 2020

patcon commented May 15, 2021

Conversation data export not working via UI #39

Conversation data export not working via UI #39

Comments

urakagi commented Oct 29, 2018 • edited by patcon Loading

To Do

urakagi commented Oct 30, 2018

metasoarous commented Oct 30, 2018

urakagi commented Oct 31, 2018

urakagi commented Oct 31, 2018 • edited Loading

metasoarous commented Oct 31, 2018

urakagi commented Oct 31, 2018

metasoarous commented Oct 31, 2018

patcon commented May 6, 2020 • edited Loading

patcon commented May 6, 2020

patcon commented Aug 15, 2020 • edited by metasoarous Loading

patcon commented Sep 18, 2020 • edited by metasoarous Loading

patcon commented Sep 18, 2020

colinmegill commented Sep 19, 2020

colinmegill commented Sep 19, 2020

colinmegill commented Sep 19, 2020

patcon commented Dec 12, 2020

patcon commented May 15, 2021

urakagi commented Oct 29, 2018 •

edited by patcon

Loading

urakagi commented Oct 31, 2018 •

edited

Loading

patcon commented May 6, 2020 •

edited

Loading

patcon commented Aug 15, 2020 •

edited by metasoarous

Loading

patcon commented Sep 18, 2020 •

edited by metasoarous

Loading