In a context where science and technology are rapidly advancing and robust solutions are needed to capture Triple Helix interactions within and across geographies, the exponential growth of scientific knowledge production remains an eminent challenge for researchers, institutions, policy-makers, funding agencies, and associated labs. The remarkable amount of R&D outputs published annually in conference proceedings, journals, and other dissemination channels has led researchers to spend less time with each paper and exclude relevant information sources to cope with specific needs. While the advances on large language models (LLMs) are being increasingly acknowledged worldwide, many challenges remain to be overcome when we transition from common sense reasoning to complex scientometric portraits characterized by volatile dynamics and heterogeneous semantic networks. Moreover, LLMs are reliant on the quality and availability of the foundational pre-trained data, resulting in problems like error propagation and hallucination. Thus, the SciCrowd project is established on the implementation of a socio-technical infrastructure able to store publication metadata classified by a crowd of users with the aim of enriching the quality of a co-evolving high-dimensional knowledge base using reinforcement learning from human feedback (RLHF). In this regard, SciCrowd is being developed 'from the ground up' as a mixed-initiative bibliometrically-enhanced information retrieval (IR) system built for (and by) researchers and the general public with the ultimate goal of conducting science and technology studies in a human-AI interactive fashion. The processes related with the analysis of basic requirements and functionalities’ specification according to functional requirements has contributed to the development of a prototype based on the mediation of the processes of scientometric output analysis, management of use levels and permissions, and data handling and visualization under different perspectives. Complementarily, a set of robustness tests and qualitative evaluations were performed to measure the system performance and end-users’ perceived usefulness and behavioral intents towards use.
SciCrowd is a crowd-powered system prototype that uses an open participation model in which humans and machines can act as "teammates" and contribute for improving the way as scientometric data is processed.
- Functional Requirements:
- [Authentication]
- [Moderation]
- [Control]
- [Interaction]
- [Visualization]
- [Security]
- Quality:
- [Usability]
- [Performance]
- [Support]
- Interfaces:
- [Human-Interfaces]
Correia, A., Grover, A., Jameel, S., Schneider, D., Antunes, P., and Fonseca, B. (2023). A hybrid human–AI tool for scientometric analysis. Artificial Intelligence Review, vol. 56, pp. 983-1010. DOI: 10.1007/s10462-023-10548-7.
Correia, A., Guimarães, D., Paredes, H., Fonseca, B., Paulino, D., Trigo, L., Brazdil, P., Schneider, D., Grover, A., and Jameel, S. (2023). NLP-crowdsourcing hybrid framework for inter-researcher similarity detection. IEEE Transactions on Human-Machine Systems, pp. 1-10. DOI: 10.1109/THMS.2023.3319290.
Correia, A., Kärkkäinen, T., Jameel, S., Schneider, D., Antunes, P., Fonseca, B., and Grover, A. (2023). A pipeline for AI-based quantitative studies of science enhanced by crowdsourced inferential modelling. In Proceedings of the 23rd International Conference on Hybrid Intelligent Systems (forthcoming).
Correia, A., Paulino, D., Paredes, H., Guimarães, D., Schneider, D., and Fonseca, B. (2023). Investigating author research relatedness through crowdsourcing: a replication study on MTurk. In Proceedings of the 2023 IEEE 26th International Conference on Computer Supported Cooperative Work in Design (CSCWD), pp. 77-82. DOI: 10.1109/CSCWD57460.2023.10152707.
Correia, A. and Lindley, S. (2022). Collaboration in relation to human-AI systems: status, trends, and impact. In Proceedings of the 2022 IEEE International Conference on Big Data (Big Data), pp. 3417-3422. DOI: 10.1109/BigData55660.2022.10020416.
Correia, A., Fonseca, B., Paredes, H., Chaves, R., Schneider, D., and Jameel, S. (2021). Determinants and predictors of intentionality and perceived reliability in human-AI interaction as a means for innovative scientific discovery. In Proceedings of the 2021 IEEE International Conference on Big Data (Big Data), pp. 3681-3684. DOI: 10.1109/BigData52589.2021.9671358.
Correia, A., Guimarães, D., Paulino, D., Jameel, S., Schneider, D., Fonseca, B., and Paredes, H. (2021). AuthCrowd: author name disambiguation and entity matching using crowdsourcing. In Proceedings of the 2021 IEEE 24th International Conference on Computer Supported Cooperative Work in Design (CSCWD), pp. 150-155. DOI: 10.1109/CSCWD49262.2021.9437769.
Correia, A., Jameel, S., Schneider, D., Paredes, H., and Fonseca, B. (2020). A workflow-based methodological framework for hybrid human-AI enabled scientometrics. In Proceedings of the 2020 IEEE International Conference on Big Data (Big Data), pp. 2876-2883. DOI: 10.1109/BigData50022.2020.9378096.
Correia, A., Jameel, S., Schneider, D., Fonseca, B., and Paredes, H. (2020). Theoretical underpinnings and practical challenges of crowdsourcing as a mechanism for academic study. In Proceedings of the 53rd Hawaii International Conference on System Sciences (HICSS), pp. 4630-4639. DOI: 10.24251/HICSS.2020.568.
Correia, A., Schneider, D., Jameel, S., Paredes, H., and Fonseca, B. (2020). Empirical investigation of the factors influencing researchers’ adoption of crowdsourcing and machine learning. In Proceedings of the 2020 20th International Conference on Intelligent Systems Design and Applications (ISDA), pp. 1257-1270. DOI: 10.1007/978-3-030-71187-0_117.
Correia, A., Fonseca, B., Paredes, H., Schneider, D., and Jameel, S. (2019). Development of a crowd-powered system architecture for knowledge discovery in scientific domains. In Proceedings of the 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC), pp. 1372-1377. DOI: 10.1109/SMC.2019.8914637.
Correia, A., Schneider, D., Paredes, H., and Fonseca, B. (2018). SciCrowd: towards a hybrid, crowd-computing system for supporting research groups in Academic Settings. In Proceedings of the 24th International Conference on Collaboration and Technology (CRIWG), pp. 34-41. DOI: 10.1007/978-3-319-99504-5_4.
Documentation: [https://link.springer.com/article/10.1007/s10462-023-10548-7]